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ABSTRACT 


This  research  paper  explores  the  use  of  performance-based  acquisition  (PBA)  techniques 
when  contracting  for  services  on  a  level-of-effort  (LOE)  basis.  The  research  explores 
how  often  and  how  effectively  these  two  concepts  are  used  together  in  the  Navy.  This 
project  examines  the  following  questions: 

1 .  Can  PBA  techniques  and  LOE  contract  types  be  used  together  effectively, 
or  is  there  a  natural  conflict  between  these  two  approaches? 

2.  Why  does  the  Navy  attempt  to  use  PBA  techniques  for  LOE  contracts? 

3.  What  are  the  consequences  of  using  PBA  techniques  for  LOE  contracts? 
An  analysis  of  50  contracts  for  services  revealed  that  the  Navy  ineffectively 

applies  PBA  techniques  to  LOE  contracts.  A  review  of  policy  and  regulations 
demonstrated  that  PBA  and  LOE  are  fundamentally  incompatible  contracting  approaches. 
The  study  suggests  possible  explanations  for  why  the  Navy  attempts  to  combine  PBA  and 
LOE  and  explores  the  consequences  of  doing  so.  This  paper  recommends  that  acquisition 
policy-makers  at  all  levels  make  clear  that  PBA  and  LOE  are  dichotomous  choices. 
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I.  INTRODUCTION 


Services  acquisition  continues  to  be  a  focus  area  for  Department  of  Defense 
(DOD)  leadership.  Ken  Brennan,  Deputy  Director,  Services  Acquisition  at  Defense 
Procurement  and  Acquisition  Policy  (DPAP),  recently  noted  that  services  account  for 
55%  of  DOD’s  acquisition  spending  (Brennan,  n.d.,  p.  3).1  The  DOD  Guidebook  for  the 
Acquisition  of  Services  notes,  “For  over  ten  years  the  DOD  has  spent  more  on  service 
requirements  than  it  has  on  equipment  acquisitions”  (Department  of  Defense  (DOD), 
2012,  p.  5).  Furthennore,  the  magnitude  of  services  acquisition  has  not  gone  unnoticed  by 
acquisition  policy-makers.  DOD’s  Better  Buying  Power  (BBP)  Initiatives  include 
“Improve  Tradecraft  in  Acquisition  of  Services”  as  one  of  seven  key  areas 
(Undersecretary  of  Defense,  Acquisition,  Technology  and  Logistics  (USD(AT&L)),  n.d.). 

PBA  is  a  key  feature  of  services  acquisition  policy — it  would  be  difficult  to 
discuss  the  topic  of  services  acquisition  without  discussing  PBA.  Current  regulations, 
policy,  and  guidance  strongly  promote  the  use  of  PBA  techniques  for  services 
acquisitions.  Services  acquisitions  that  use  PBA  techniques  are  called  performance-based 
service  acquisitions  (PBSA).  The  underlying  rationale  of  PBA  is  that  the  government 
should  not  tell  contractors  how  to  perform,  because  doing  so  would  stifle  industry’s 
creativity.  Instead,  the  government  should  define  its  requirements  in  terms  of  what 
outcomes  the  contractor  needs  to  accomplish  without  specifying  the  details.  Such  an 
approach,  its  proponents  argue,  improves  competition  and  empowers  private  industry  to 
innovate  and  accomplish  desired  objectives  more  efficiently.  PBA’s  rationale  is  well- 
established  within  the  federal  government  and  is  reflected  in  current  regulations,  policy, 
and  guidance.  There  is  a  clear  preference  for  using  PBA  methods  for  the  acquisition  of 
services  in  Federal  Acquisition  Regulations  (FAR)  37.102(a). 

The  policy  push  for  PBA  has  made  its  way  into  the  Navy  acquisition  culture.  In 
fact,  until  19  March  2015,  the  Navy  required  approvals  above  the  contracting  officer’s 
level  for  services  acquisitions  not  classified  as  PBSA  (approval  requirements  were 

1  DPAP  is  an  office  under  the  Undersecretary  of  Defense,  Acquisition,  Technology,  and  Logistics 
(USD(AT&L)). 
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recently  removed  per  Navy  and  Marine  Corps  Acquisition  Regulation  Supplement 
(NMCARS)  Change  13-05).  However,  there  is  a  large  category  of  services  acquisitions 
that  does  not  mesh  well  with  PBA’s  rationale — services  acquired  on  the  basis  of  a  level 
of  effort.  Generally,  services  contracts  define  the  contractor’s  obligation  in  one  of  two 
ways:  by  describing  the  required  work  in  terms  of  the  completion  of  one  or  more 
specified  tasks  (“completion  contracts”)  or  by  describing  the  required  amount  of  effort 
(typically  measured  in  labor  hours)  that  the  contractor  must  expend  in  performing  one  or 
more  specified  tasks  (“level-of-effort  (LOE)”  contracts)  (Cibinic,  Nash  Jr.,  &  Yukins, 
2011,  p.  1317).  There  is  a  regulatory  preference  to  use  the  completion  type,  because  it 
contractually  obligates  the  contractor  to  produce  an  end  product  or  result.  However,  the 
regulations  recognize  that  not  all  requirements  can  state  a  definite  goal  or  target,  or 
specify  an  end  result.  This  is  why  a  second  type  exists,  the  LOE  type.  LOE  includes  time- 
and-materials  (T&M)  contracts,  labor-hour  (LH)  contracts,  cost-reimbursement  tenn 
contracts,  and  firm-fixed-price  LOE  term  contracts  (FFP-LOE)  (Cibinic,  Nash  Jr.,  & 
Yukins,  201  l,p.  1318). 

There  is  an  apparent  contradiction  between  PBA  and  LOE:  If  one  cannot  define 
the  work  in  tenns  of  a  definite  goal,  target,  or  end  product,  thus  suggesting  the  use  of  an 
LOE  contract  type,  one  cannot  establish  meaningful  performance  outcomes  as  required 
under  PBA. 

This  research  paper  explores  the  use  of  PBA  techniques  when  contracting  for 
services  on  an  LOE  basis  by  examining  three  central  research  questions.  Because  each 
central  research  question  is  complex,  Chapter  IV  of  this  paper  analyzes  each  by  exploring 
several  sub-questions.  These  sub-questions  lend  to  a  more  thorough  analysis  of  the  PBA- 
LOE  relationship.  The  central  research  questions  and  sub-questions  follow: 

•  Question  #1:  Can  PBA  techniques  and  LOE  contract  types  be  used 
together  effectively,  or  is  there  a  natural  conflict  between  these  two 
approaches? 

First,  we  first  need  to  understand  the  relationship  between  PBA  and  LOE. 
We  need  to  know  how  often  and  how  effectively  the  Navy  uses  these  two 
approaches  together. 
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Question  #1  a:  Do  regulations  and  policy  create  a  distinct  choice  between 
PBA  and  LOE? 

Question  #lb:  Is  the  dichotomous  relationship  between  PBA  and  LOE 
rational? 

Question  #1  c:  Is  the  dichotomous  relationship  between  PBA  and  LOE 
apparent  in  the  data? 

•  Question  #2:  Why  does  the  Navy  attempt  to  use  PBA  techniques  for  LOE 
contracts? 

After  understanding  the  relationship  between  PBA  and  LOE,  we  can 
further  explore  the  topic  by  examining  possible  motivations  for  combining 
the  two  approaches. 

Question  #2a:  Does  the  history  of  PBA  policy  initiative  explain  why  the 
Navy  combines  PBA  and  LOE? 

Question  #2b:  What  incentives  does  the  PBA  policy  initiative  give 
contracting  activities? 

Question  #2c:  Does  literature  on  PBA  make  clear  that  it  does  not  apply  to 
LOE? 

•  Question  #3:  What  are  the  consequences  of  using  PBA  techniques  for 
LOE  contracts? 

Finally,  we  need  to  explore  what  happens  when  these  two  approaches  are 
combined. 

Question  #3a:  What  kind  of  performance  incentives  are  created  when  PBA 
is  applied  to  LOE,  and  what  impacts  do  these  incentives  have? 

Question  #3b:  What  administrative  burden  is  involved  with  mixing  PBA 
and  LOE? 

Question  #3c:  What  happens  when  PBA  is  not  used  with  LOE? 

Exploring  these  questions  required  a  thorough  review  of  statutes,  regulations, 
policy,  guidance,  and  scholarly  articles  regarding  PBA.  Even  a  quick  Internet  search 
reveals  that  there  is  rich  literature  on  this  topic.  Much  can  be  understood  about  the 
relationship  between  PBA  and  LOE  by  applying  logical  argumentation  from  the  premises 
of  policy.  However,  analyzing  policy  and  literature  alone  is  not  sufficient — a  more 
complete  understanding  of  how  the  Navy  combines  PBA  and  LOE  may  be  obtained 
through  an  examination  of  actual  contracts  for  services. 
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This  research  examined  50  Navy  contracts  for  services  that  were  identified  as 
PBSA,  detennined  whether  these  contracts  were  completion  or  LOE,  then  evaluated  the 
LOE  contracts  more  closely  to  determine  whether  PBA  was  properly  applied. 
Furthennore,  the  researcher  noted  observations  regarding  the  supposed  perfonnance 
results,  acceptable  quality  levels,  and  planned  methods  for  evaluating  performance. 

This  paper  describes  the  method  for  selecting  and  evaluating  the  sample,  presents 
the  data,  then  analyzes  the  three  central  research  questions  and  their  sub-questions  within 
the  context  of  the  data,  relevant  policy  and  literature,  and  logical  argumentation  from  the 
premises  of  policy.  Ultimately,  this  paper  draws  several  conclusions  regarding  the 
relationship  between  PBA  and  LOE,  including  the  motivations  behind  and  consequences 
of  combining  the  two  approaches.  The  study’s  conclusions  inform  specific 
recommendations  for  improving  LOE  contracts  for  services. 

This  study  has  four  noteworthy  limitations.  First,  only  Navy  and  Marine  Corps 
contracts  were  included  in  the  sample.  This  research  does  not  speculate  how  often  other 
government  or  DOD  agencies  combine  PBA  and  LOE.  However,  the  analysis  of  the 
natural  conflict  between  PBA  and  LOE  is  based  primarily  on  rules  and  policies  that  apply 
to  all  government  agencies;  although  the  study  cannot  conclude  that  other  agencies 
combine  the  concepts,  it  can  conclude  that  such  a  combination  would  be  improper. 
Second,  only  contracts  awarded  between  1  March  2014  and  1  March  2015  were  included 
in  the  sample.  This  study  cannot  draw  conclusions  related  to  the  time  period  prior  to  1 
March  2014  based  on  the  sampled  contracts  alone.  However,  other  sources  can  support 
inferences  and  conclusions  about  the  history  of  combining  PBA  and  LOE.  Third,  the 
study  is  limited  to  the  application  of  PBA  to  services  acquired  on  an  LOE  basis.  The 
study  does  not  make  conclusions  regarding  the  effectiveness  of  PBA  in  other  types  of 
contracts.  Finally,  when  this  paper  discusses  the  use  of  PBA  and  LOE  together,  it 
assumes  that  both  approaches  are  being  applied  to  the  same  task.  This  study  does  not 
address  (or  discount)  the  approach  of  using  both  PBA  and  LOE  in  the  same  contract  yet 
for  different  tasks.  In  theory,  a  contract  could  include  several  separate  tasks,  some  of 
which  are  suitable  for  the  LOE  form  and  others  that  are  suitable  for  the  completion  form. 
These  tasks  could  be  separated  by  contract  line  items;  different  contracting  approaches 
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could  then  be  applied  to  each  line  item  based  on  suitability.  Although  the  researcher  did 
not  observe  any  such  instances  in  the  sample,  this  approach  was  not  studied  directly. 
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II.  METHOD  FOR  SELECTING  AND  EVALUATING  THE 

SAMPLE 

A.  CHAPTER  DESCRIPTION 

This  chapter  describes  the  method  for  selecting  and  evaluating  the  sample. 
Because  the  focus  of  this  research  is  PBA’s  applicability  to  LOE,  this  research  is  only 
concerned  with  contracts  that  are  identified  as  both  PBSA  and  LOE.  Upon  selecting  a 
relevant  sample,  the  study  followed  a  structured  process  for  evaluating  each  sampled 
contract.  This  chapter  describes  the  step-by-step  method  used  to  select  and  evaluate  the 
sample.  The  results  of  this  evaluation  are  contained  in  Chapter  III. 

B.  METHOD  FOR  SELECTING  THE  SAMPLE 

For  the  purpose  of  answering  the  research  questions,  a  relevant  sample  for 
evaluation  includes  only  those  contracts  that  are  identified  as  both  PBSA  and  LOE. 
Several  steps  were  required  to  establish  a  relevant  sample  for  evaluation.  First,  the 
researcher  obtained  a  list  of  all  contracts  containing  the  following  characteristics  from  the 
Federal  Procurement  Data  System  (FPDS): 

1 .  Contract  or  task  order  (excluding  grants,  cooperative  agreements,  or 
technology  agreements); 

2.  Issued  by  the  Navy  or  Marine  Corps  (excluding  contracts  that  the  Navy 
and  Marine  Corps  issued  under  non-Navy  contracts); 

3.  For  services  (Product  Service  Code  beginning  in  alpha  character,  rather 
than  Federal  Supply  Code  beginning  in  numeric  character); 

4.  Issued  between  1  March  2014  and  1  March  2015;  and 

5.  Indicates  “Yes  -  Service  where  PBA  is  used”  in  the  “Performance  Based 
Service  Acquisition”  field.2 

Second,  the  researcher  refined  the  raw  data  to  focus  on  LOE  contracts.  This  was 
necessary  because,  unfortunately,  FPDS  does  not  include  a  separate  data  field  for  LOE. 
For  purposes  of  this  study,  three  contract  types  are  LOE  by  definition:  T&M,  LH,  and 


2  This  field  is  manually  selected  by  the  contracts  specialist  in  FPDS. 
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FFP-LOE.  Cost-reimbursement  contracts  may  be  structured  as  LOE.  However,  because 
FPDS  does  not  clearly  identify  LOE  as  a  separate  reporting  field,  the  raw  FPDS  data  does 
not  distinguish  between  a  cost-reimbursement  completion  type  contract  and  a  cost- 
reimbursement  LOE  type  contract.  Therefore,  no  cost-reimbursement  type  contracts  were 
excluded  from  the  study  at  this  phase.  The  detennination  of  whether  each  sampled  cost- 
reimbursement  contract  was  LOE  or  completion  required  evaluation  of  the  contract  itself. 
This  process  will  be  discussed  later  in  this  chapter. 

The  researcher  removed  non-LOE  contract  types  from  the  list  of  raw  data;  the 
data  now  comprised  the  entire  population  of  contracts  marked  as  PBSA  that  could  be 
LOE. 

Finally,  the  researcher  chose  a  sample  of  50  contracts  from  the  population  of 
4,785  contracts.  The  sample  size  of  50  was  chosen  not  to  establish  statistical  significance, 
but  rather  to  sufficiently  investigate  trends.  The  sample  was  chosen  randomly  using 
Microsoft  Excel.  The  random  sample  represents  1.04  percent  of  the  total  population  of 
contracts  for  services  issued  by  the  Navy  or  Marine  Corps  that  are  marked  as  PBSA  and 
could  be  LOE.  The  list  of  sampled  contracts,  entitled  “Random  Sample,”  is  included  as 
Appendix  A. 

C.  METHOD  FOR  EVALUATING  THE  SAMPLE 

The  sample  was  evaluated  through  an  established  and  objective  process.  This 
process  is  depicted  in  Figure  1. 
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STEP  #1: 


If  completion  or  not  clear, 
contract  not  further  analyzed. 


Identify  all  cost-reimbursement  contracts 
as  LOE,  completion,  or  not  clear. 


Figure  1 .  Process  for  evaluating  the  sample. 


An  explanation  of  the  specific  steps  in  Figure  1  follows. 

In  Step  #1,  the  researcher  identified  all  cost-reimbursement  contracts  as  either 
LOE  or  completion.  As  discussed  previously,  T&M,  LH,  and  FFP-LOE  contract  types 
were  considered  LOE  by  definition,  but  cost-reimbursement  contracts  may  be  either 
completion  or  LOE.  The  data  fields  in  FPDS  do  not  distinguish  between  LOE  and 
completion  for  cost-reimbursement  contracts,  so  evaluation  of  the  individual  contracts 
was  necessary. 
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The  criteria  used  for  identifying  a  contract  as  LOE  or  completion  were  taken  from 
the  FAR’s  definitions  of  these  types.  FAR  16.306(d)(1)  states,  “a  completion  fonn 
describes  the  scope  of  work  by  stating  a  definite  goal  or  target  and  specifying  an  end 
product”  (Federal  Acquisition  Regulation  [FAR],  2015).  Contracts  meeting  this  definition 
were  recorded  as  completion.  FAR  16.306(d)(2)  states,  “a  tenn  form  describes  the  scope 
of  work  in  general  terms  and  obligates  the  contractor  to  devote  a  specified  level  of  effort 
for  a  stated  time  period”  (FAR,  2015).  Contracts  meeting  this  definition  were  classified 
as  LOE. 

Some  cost-reimbursement  contracts  analyzed  could  not  be  classified  as  either 
completion  or  LOE;  these  contracts  were  recorded  as  “not  clear.”  This  was  due  either  to 
lack  of  access  to  the  contract  specification  (e.g.,  the  performance  work  statement  (PWS) 
or  statement  of  work  (SOW)),  or  an  apparent  intermingling  of  the  two  distinct  contract 
types.  Several  contracts  referenced  a  specification  as  a  contract  attachment,  but  this 
attachment  was  not  available  in  the  Electronic  Document  Access  (EDA)  system.  It  is  not 
possible  to  detennine  conclusively  whether  a  cost-reimbursement  contract  is  completion 
or  LOE  without  reviewing  the  specification,  so  such  contracts  were  recorded  as  “not 
clear.”  Other  contracts  intermingled  the  two  distinct  fonns.  Some  identified  a  definite 
goal  or  target,  but  also  obligated  the  contractor  to  perfonn  a  specified  level  of  effort. 
Other  contracts  used  both  “completion”  and  “LOE”  throughout  the  contract,  as  if  the 
contract  could  simultaneously  include  both  forms  on  a  single  contract  line  item.  If  the 
contract  did  not  clearly  meet  the  definition  of  LOE  or  completion,  it  was  recorded  as  “not 
clear.” 

Because  this  research  focuses  on  LOE  contracts,  contracts  that  were  identified  as 
completion  or  “not  clear”  were  not  further  analyzed. 

The  researcher  then  evaluated  four  specific  criteria  to  determine  whether  PBA 

was  applied  correctly  (Steps  #2-#5  in  Figure  1).  The  central  question  of  the  research  is 

“Can  PBA  techniques  and  LOE  contract  types  be  used  together  effectively,  or  is  there  a 

natural  conflict  between  these  two  approaches?”  To  explore  this  question,  the  researcher 

evaluated  whether  the  Navy  applied  PBA  techniques  properly  to  each  LOE  contract  in 

the  sample.  If  the  sample  demonstrated  that  PBA  techniques  were  applied  properly  to 
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LOE  contracts,  this  might  suggest  that  PBA  techniques  are  appropriate  for  LOE  services. 
Conversely,  if  the  sample  demonstrated  that  PBA  techniques  were  not  applied  properly  to 
LOE  contracts,  this  might  suggest  that  PBA  techniques  are  not  appropriate  for  LOE 
services. 

The  researcher  recognizes  the  need  to  define  “proper” — how  does  one  know  if 
PBA  techniques  were  “properly”  applied?  For  the  purposes  of  this  research,  the 
researcher  assumes  that  PBA  techniques  were  properly  applied  if  the  contract  meets  the 
FAR’s  definition  of  PBSA  and  meets  any  additional  minimum  standards  imposed  by  the 
DOD  Guidebook  for  the  Acquisition  of  Services. 

For  the  purposes  of  this  study,  the  minimum  criteria  used  to  determine  if  PBA 
techniques  were  properly  applied  are  as  follows:3 

1 .  The  PWS  describes  a  required  result  rather  than  either  “how”  the  work  is 
to  be  accomplished  or  the  number  of  hours  to  be  provided  (FAR 
37.602(b)(1)).  This  was  evaluated  as  Step  #2  of  the  process; 

2.  The  contract  shall  include  measureable  perfonnance  standards  (FAR 
37.601(b)(2)).  This  was  evaluated  as  Step  #3  of  the  process; 

3.  The  contract  shall  include  a  method  for  assessing  performance  against 
perfonnance  standards  (FAR  37.601(b)(2)).  This  was  evaluated  as  Step  #4 
of  the  process;  and 

4.  The  contractor’s  performance  against  the  required  standards  must  be 
measureable  through  an  objective  process  (DOD  Guidebook  for  the 
Acquisition  of  Services,  p.  9).4  This  was  evaluated  as  Step  #5  of  the 
process.5 


3  Further  information  regarding  how  the  researcher  evaluated  each  of  these  standards  is  included  as 
Appendix  B,  entitled,  “Detailed  Evaluation  Criteria  for  Determining  Proper  Application  of  PBA.” 

4  Although  FAR  part  37  requires  only  that  the  performance  standards  are  measureable,  DOD  requires 
that  these  standards  are  measureable  through  an  objective  process.  Furthermore,  objectivity  is  discussed  in 
the  definition  of  a  PWS  in  FAR  2.101. 

5  The  researcher  notes  that  there  is  a  requirement  at  FAR  37.601(b)(1)  to  include  a  PWS  in 
performance-based  contracts.  Flowever,  there  is  no  requirement  to  label  a  PWS  as  such;  therefore, 
evaluating  the  title  of  the  specification  would  be  meaningless.  A  PWS  is  defined  in  FAR  2.101  as  “a 
statement  of  work  for  performance-based  acquisitions  that  describes  the  required  results  in  clear,  specific 
and  objective  terms  with  measurable  outcomes”  (FAR,  2015).  This  definition  is  covered  by  the  other 
criteria  evaluated  in  this  study,  therefore  the  inclusion  of  a  PWS  was  not  evaluated  as  a  distinct  criterion. 
The  researcher  also  notes  that  an  additional  standard  exists  at  FAR  37.601(b),  “Performance-based 
contracts  for  services  shall  include  -...(3)  Performance  incentives  where  appropriate”  (FAR,  2015). 
Flowever,  this  was  not  considered  in  this  study,  because  the  appropriateness  of  incentives  is  subjective. 
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In  accordance  with  Steps  #2-#5  in  Figure  1,  each  LOE  contract  in  the  sample  was 
analyzed  to  determine  whether  it  satisfied  each  of  these  minimum  criteria.  The  criteria 
evaluated  for  each  sample  contract  is  also  represented  in  Appendix  C,  entitled  “Blank 
Data  Sheet.”  The  researcher  populated  this  spreadsheet  for  all  contracts  sampled,  in 
accordance  with  Figure  1 . 

In  Step  #6  of  the  evaluation,  the  researcher  detennined  whether  PBA  was 
properly  applied.  If  the  contract  met  all  the  minimum  criteria,  it  was  determined  that  PBA 
was  properly  applied.  If  the  contract  did  not  meet  all  the  minimum  criteria,  it  was 
detennined  that  PBA  was  not  properly  applied.  Even  if  some  but  not  all  of  the  minimum 
criteria  were  met,  it  was  still  detennined  that  PBA  was  not  properly  applied.  Based  on  the 
standards  listed  in  the  FAR  and  the  DOD  Guidebook  for  the  Acquisition  of  Services,  all 
minimum  criteria  must  be  met  to  be  defined  as  PBA. 

Throughout  the  six-step  process,  the  researcher  also  noted  observations  regarding 
the  perfonnance  results,  acceptable  quality  levels,  and  planned  methods  for  evaluating 
perfonnance.  Although  numerical  data  can  help  answer  the  research  questions, 
observations  from  the  sample  are  necessary  to  understand  the  relationship  between  PBA 
and  LOE  fully.  For  example,  the  numerical  data  may  tell  us  how  often  the  Navy  properly 
applied  PBA  techniques  to  LOE  services  contracts,  yet  it  cannot  tell  us  what  a  contract 
looks  like  when  PBA  and  LOE  are  combined.  The  numerical  data  alone  cannot  answer 
why  the  Navy  applies  PBA  to  LOE  contracts  or  describe  the  consequences  of  doing  so. 
With  the  research  questions  in  mind,  the  researcher  noted  several  trends  during  the 
evaluation. 

In  summary,  this  research  examined  50  Navy  contracts  for  services  that  were 
identified  as  PBS  A,  then  objectively  evaluated  LOE  contracts  more  closely  to  determine 
whether  PBA  was  properly  applied.  Furthermore,  the  researcher  noted  observations 
regarding  the  perfonnance  results,  acceptable  quality  levels,  and  planned  methods  for 
evaluating  perfonnance.  The  numerical  data  and  observations  produced  by  this  research 
process  informed  the  analysis  of  the  research  questions. 
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III.  THE  DATA 


This  chapter  presents  the  numerical  data  that  were  obtained  using  the  detailed 
process  described  in  Chapter  II.  The  data  are  presented  simply  as  they  were  observed, 
without  analysis.  The  relevance  of  the  data  to  the  research  questions  will  be  discussed  in 
Chapter  IV. 

This  chapter  does  not  list  the  researcher’s  observations  regarding  performance 
results,  acceptable  quality  levels,  and  planned  methods  for  evaluating  performance. 
Individual  observations  will  be  introduced  as  they  apply  to  specific  research  questions  in 
Chapter  IV. 

The  data  spreadsheet  was  populated  then  sorted  to  group  LOE  contracts.  The 
sorted  data  spreadsheet  is  included  as  Appendix  D,  entitled,  “Completed  Data  Sheet.” 
Results  are  presented  in  the  same  order  as  the  steps  identified  in  Figure  1. 

During  Step  #1  of  the  research  process,  22  of  the  50  contracts  sampled  were 
identified  as  LOE.  Nine  of  50  contracts  were  identified  as  completion.  LOE  or 
completion  could  not  be  clearly  identified  in  19  of  50  contracts  sampled.  This  is  depicted 
in  Figure  2. 


Figure  2. 


Not  Clear,  19, 
38% 


Total  LOE,  22, 
44% 


Completion,  9, 
18% 


Percentages  of  LOE,  Completion,  and  Not  Clear 


13 


In  Step  #2a  of  the  process,  the  researcher  found  that  21  of  22  LOE  contracts 
sampled  did  not  specify  a  performance  result.  This  is  depicted  in  Figure  3. 


Yes,  1,  5% 


No,  21,  95% 

Figure  3.  Does  the  contract  specify  a  performance  result? 

In  Step  #2b  of  the  process,  the  researcher  found  that  22  of  22  LOE  contracts  sampled 
specified  a  required  number  of  hours  to  be  provided.  This  is  depicted  in  Figure  4. 


No,  o,  o% _ 


Yes,  22,  100% 


Figure  4.  Does  the  contract  specify  a  number  of  hours  to  be  provided? 
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In  Step  #3  of  the  process,  the  researcher  found  that  15  of  22  LOE  contracts 
sampled  did  not  include  measurable  performance  standards.  Additionally,  it  was  not  clear 
whether  measureable  performance  standards  existed  in  5  of  the  22  LOE  contracts 
sampled.  This  is  depicted  in  Figure  5. 


Yes,  2,  9% 


Not  Clear,  5, 
23% 


NO,  15,  68% 


Figure  5.  Does  the  contract  include  measureable  standards? 


In  Step  #4  of  the  process,  1  of  22  LOE  contracts  sampled  did  not  include  a 
method  by  which  to  measure  perfonnance  against  the  standards.  Additionally,  it  was  not 
clear  whether  this  criterion  was  met  for  19  of  the  22  LOE  contracts  sampled.  As 
discussed  in  Appendix  B,  this  was  mainly  due  to  inaccessibility  of  the  Quality  Assurance 
Surveillance  Plan  (QASP);  this  does  not  demonstrate  a  significant  finding.  This  is 
depicted  in  Figure  6. 


Not  Clear,  19, 
86% 


Figure  6.  Does  the  contract  include  a  method  of  assessing  performance? 
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In  Step  #5  of  the  process,  1  of  22  LOE  contracts  sampled  did  not  provide  an 
objective  process  by  which  to  measure  performance.  Additionally,  it  was  not  clear 
whether  this  criterion  was  met  for  19  of  the  22  LOE  contract  sampled.  As  discussed  in 
Appendix  B,  this  was  mainly  due  to  inaccessibility  of  the  QASP;  this  does  not 
demonstrate  a  significant  finding.  This  is  depicted  in  Figure  7. 


Figure  7.  Performance  can  be  measured  through  an  objective  process? 

In  Step  #6,  the  final  step  of  the  process,  the  researcher  found  that  none  of  the  LOE 
contracts  sampled  properly  applied  all  PBA  minimum  criteria.  This  is  depicted  in  Figure  8. 


Yes,  0,  0% 


No,  22,  100% 

Figure  8.  PBA  techniques  properly  applied? 
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The  evaluation  of  the  sampled  contracts  discovered  that  not  a  single  LOE  contract 
sampled  met  all  the  minimum  criteria  for  the  proper  use  of  PBA.  Although  some 
contracts  included  measureable,  objective  methods  of  evaluating  perfonnance,  these 
methods  were  not  tied  to  performance  results.  Notably,  not  a  single  LOE  contract 
specified  a  performance  result  rather  than  specifying  a  number  of  hours  to  be  provided. 
The  data  presented  in  this  chapter  will  inform  the  analysis  of  the  research  questions. 
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IV.  ANALYSIS 


A.  CHAPTER  INTRODUCTION 

The  data  suggest  that  the  Navy  does  not  properly  apply  PBA  techniques  to  LOE 
contracts.  Not  a  single  LOE  contract  sampled  met  the  minimum  criteria  for  the  proper  use 
of  PBA  techniques.  If  PBA  techniques  are  effectively  applied  to  LOE  contracts,  one 
would  expect  to  find  at  least  one  such  instance  in  the  sample.  This  leads  back  to  the  three 
central  research  questions: 

•  Can  PBA  techniques  and  LOE  contract  types  be  used  together  effectively, 
or  is  there  a  natural  conflict  between  these  two  approaches? 

•  Why  does  the  Navy  attempt  to  use  PBA  techniques  for  LOE  contracts? 

•  What  are  the  consequences  of  using  PBA  techniques  for  LOE  contracts? 

This  chapter  analyzes  the  three  central  research  questions  and  their  sub-questions 
within  the  context  of  the  data,  relevant  policy  and  literature,  and  logical  argumentation 
from  the  premises  of  policy. 

B.  THE  CONFLICT  BETWEEN  PBA  AND  LOE 

Question  #1  of  the  research  is:  Can  PBA  techniques  and  LOE  contract  types  be 
used  together  effectively,  or  is  there  a  natural  conflict  between  these  two  approaches? 
The  analysis  of  this  question  is  divided  into  three  parts.  The  relationship  between  PBA 
and  LOE  is  examined  first  in  the  context  of  regulations  and  policy,  then  from  a  rational 
perspective,  and  finally  using  the  research  data.  The  analysis  reveals  that  PBA  and  LOE 
are  two  naturally  conflicting  approaches  that  cannot  be  used  together  effectively. 

1.  Regulations  and  Policy  Create  a  Distinct  Choice 

Question  #la  of  the  research  is:  Do  regulations  and  policy  create  a  distinct  choice 
between  PBA  and  LOE?  The  most  direct  citation  illustrating  the  PBA-LOE  conflict  is  at 
FAR  37.602(b).  This  states: 
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Agencies  shall,  to  the  maximum  extent  practicable  -  (1)  Describe  the  work 
in  terms  of  the  required  results  rather  than  either  “how”  the  work  is  to  be 
accomplished  or  the  number  of  hours  to  be  provided.  (FAR,  2015) 

This  creates  a  distinct  choice.  Work  must  be  described  either  a)  in  tenns  of  the 
required  results,  or  b)  either  “how”  the  work  is  to  be  accomplished  or  the  number  of 
hours  to  be  provided.  In  other  words,  work  cannot  be  described  in  both  of  these  ways. 
Using  PBA  requires  the  use  of  a  PWS  (FAR  37.601(b)(1)),  which  by  definition  describes 
the  work  in  terms  of  required  results  (FAR  2.101).  Using  an  LOE  approach,  by  definition, 
describes  the  work  in  terms  of  number  of  hours  to  be  provided  (Cibinic,  Nash  Jr.,  & 
Yukins,  2011,  p.  1317).  Therefore,  applying  FAR  37.602(b),  it  logically  follows  that 
these  two  approaches  cannot  be  used  simultaneously. 

Although  FAR  37.602(b)(1)  creates  this  distinct  choice,  this  was  apparently  not 
the  intent  of  the  policy-makers  who  created  this  regulation.  In  July  2004,  a  proposed  rule 
regarding  PBSA  was  published  in  the  Federal  Register  (FR)  at  69  FR  43712.  This 
proposed  rule  included  an  invitation  for  public  comments;  responses  to  comments  were 
published  at  71  FR  211  in  January  2006.  One  of  these  comments  directly  addressed  the 
PBA-LOE  conflict: 

One  commenter  said  the  rule  should  contain  a  strong  statement  to 
emphasize  that  performance-based  contracting  requires  an  end  product  or 
service  that  can  be  measured  and  that  labor  hour  instruments  are  level-of- 
effort  contracts  with  no  definite  deliverable.  (71  Federal  Register  (FR) 

211) 

The  Councils  addressed  this  comment  with  the  following: 

By  definition,  all  contracts  require  delivery  of  supplies  or  performance  of 
services.  The  deciding  factor  for  perfonnance-based  acquisitions  is 
whether  or  not  the  contract  has  measurable  performance  standards.  The 
Councils6  believe  that  T&M/LH  contracts  can  have  measurable 
perfonnance  standards.  Therefore,  the  rule  does  not  preclude  the  use  of 
T&M/LH  contracts  for  performance-based  acquisitions.  (71  Federal 
Register  (FR)  211) 


6  The  term  “Councils”  refers  to  the  Civilian  Agency  Acquisition  Council  and  the  Defense  Acquisition 
Regulations  (DAR)  Council,  the  rule-making  authorities  for  federal  acquisition  regulations. 
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By  stating  here  that  PBA  can  apply  to  LOE  contracts,  it  appears  that  the  distinct  choice 
created  by  FAR  37.602(b)  was  unintentional. 

However,  the  Councils’  position  is  contradictory.  Although  the  Councils  stated 
that  the  “deciding  factor  for  perfonnance-based  acquisitions  is  whether  or  not  the  contract 
has  measurable  performance  standards,”  the  regulations  do  not  support  this  position.  As 
discussed  in  Chapter  II,  the  existence  of  measureable  perfonnance  standards  is  only  one 
of  several  conditions  that  must  be  present  to  meet  the  regulatory  definition  of  PBA.  The 
regulations  do  not  allow  the  use  of  any  single  condition  as  a  “deciding  factor.” 
Furthennore,  although  the  Councils  believed  that  LOE  contracts  can  have  measureable 
perfonnance  standards,  they  did  not  support  this  belief  with  further  explanation  or 
evidence.7  It  seems  that  the  flaws  in  the  Councils’  position  have  since  been  recognized. 
Recently,  policy-makers  at  Office  of  the  Undersecretary  of  Defense,  Acquisition, 
Technology  and  Logistics  (OUSD  (AT&L))  have  clarified  that  PBA  and  LOE  are  distinct 
choices.  The  dichotomous  relationship  between  PBA  and  LOE  is  addressed  in 
OUSD(AT&L)’s  memorandum  entitled  “Guidelines  for  Creating  and  Maintaining  a 
Competitive  Environment  for  Supplies  and  Services  in  the  Department  of  Defense,” 
issued  in  August  2014.  This  document  states: 

Identify  the  desired  output  in  perfonnance-based  terms  if  possible;  if  not 
practical  for  a  particular  requirement,  then  establish  level-of-effort  terms 
for  that  particular  requirement  solely.  (Office  of  the  Undersecretary  of 
Defense,  Acquisition,  Technology  and  Logistics  (OUSD(AT&L),  2014,  p. 

17) 

This  statement  aligns  with  FAR  37.602(b);  contracting  officers  must  choose  either 
PBA  or  LOE,  but  not  both. 

OUSD(AT&L)  also  discusses  this  dichotomous  relationship  in  a  memorandum 
entitled,  “Appropriate  Use  of  Lowest  Price  Technically  Acceptable  Source  Selection 
Process  and  Associated  Contract  Type,”  issued  4  March  2014.  The  document  states: 

In  some  cases,  our  requirements  are  firm,  easily  understood,  and  tied  to 
clear,  measureable  outcomes. .  .In  other  cases,  we  cannot  firmly  predict  the 
tasks,  efforts,  and  required  outcomes...  This  situation  lends  itself  to  CPFF 

7  The  challenges  and  effects  of  combining  PBA  and  LOE  are  discussed  later  in  this  chapter. 
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LOE...or  in  some  circumstances  a  T&M  contract  type.”  (OUSD(AT&L), 

2014,  p.  2) 

It  is  important  to  note  OUSD(AT&L)’s  use  of  the  phrase  “clear,  measureable 
outcomes.”  These  are  pre-requisites  for  using  PBA.  Therefore,  the  guidance  reiterates 
that  if  the  government  cannot  tie  the  requirement  to  clear,  measureable  outcomes,  then 
PBA  is  not  suitable  for  the  requirement  and  an  LOE  contract  type  is  more  appropriate. 

Notwithstanding  the  Councils’  original  intent,  FAR  37.602(b)(1)  still  creates  a 
distinct  choice  between  PBA  and  LOE.  Modern  policy-makers  at  OUSD(AT&L)  seem  to 
agree  that  the  relationship  between  PBA  and  LOE  is  dichotomous. 

2.  The  Dichotomous  Relationship  is  Rational 

Question  #lb  of  the  research  is:  Is  the  dichotomous  relationship  between  PBA  and 
LOE  rational?  This  question  requires  setting  aside  the  regulations  (only  for  a  moment) 
and  analyzing  the  relationship  between  PBA  and  LOE  from  a  rational  perspective.  Such 
an  analysis  reveals  that  the  two  concepts  are,  based  on  their  underlying  rationales, 
incompatible. 

The  underlying  rationale  of  PBA  is  that  by  describing  the  work  in  clear,  specific, 
and  objective  terms  with  measureable  outcomes,  the  government  can  focus  the 
contractor’s  attention  on  desired  outcomes  rather  than  “how  to”  details  (which  are 
presumably  not  important).  This  approach,  its  proponents  argue,  unleashes  private 
industry’s  creativity,  resulting  in  both  higher  quality  perfonnance  and  cost  savings.  Other 
supposed  benefits  include  maximizing  competition,  promoting  the  use  of  commercial 
services,  and  shifting  risk  from  government  to  industry  (DOD,  2012,  p.  8-9).  In  theory, 
the  government  can  establish  the  perfonnance  outcomes  then  step  away  and  let  the 
contractor  perfonn.  This  rationale  relies  on  the  assumption  that  the  requirement  can  be 
defined  in  terms  of  clear,  specific,  and  objective  terms  with  measureable  outcomes. 
However,  if  the  results  are  not  clear,  how  does  the  contractor  know  what  to  do?  How  can 
the  government  step  away  and  allow  the  contractor  to  innovate?  How  does  the 
government  objectively  measure  performance  against  results  if  the  results  are  not  well- 
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defined  to  begin  with?  Without  defining  clear,  specific,  and  objective  terms  with 
measureable  outcomes,  the  theoretical  benefits  of  the  PBA  approach  are  lost. 

LOE  cannot  meet  these  necessary  conditions.  According  to  Formation  of 
Government  Contracts,  the  LOE  contract  type  is  “used  sparingly  since  it  imposes 
significantly  less  risk  on  the  contractor  than  contracts  calling  for  completion  of  a 
specified  task”  (Cibinic,  Nash  Jr.,  &  Yukins,  2011,  p.  1317).  The  contracting  officer 
cannot  use  LOE  unless  specific  conditions  are  met: 

•  The  cost-reimbursement  term  fonn  can  only  be  used  when  the  requirement 
cannot  be  defined  well  enough  to  pennit  the  development  of  estimates 
within  which  the  contractor  can  be  expected  to  complete  the  work  (FAR 
16.306(d)(2));  or 

•  The  T&M  and  LH  contract  types  can  only  be  used  when  it  is  not  possible 
to  estimate  accurately  the  extent  or  duration  of  the  work  or  to  anticipate 
costs  with  any  reasonable  degree  of  confidence  (FAR  16.601(c));  or 

•  The  FFP-LOE  tenn  contract  type  can  only  be  used  when  the  work  required 
cannot  otherwise  be  clearly  defined.  (FAR  16.207-3(a)) 

When  a  contracting  officer  has  determined  that  LOE  is  appropriate,  he/she  has 
detennined  that  one  of  these  conditions  is  present.  This  raises  the  question:  If  a 
requirement  is  so  unclear  that  it  meets  these  conditions,  how  can  it  be  considered  to 
contain  “clear,  specific  and  objective  terms  with  measureable  outcomes,”  as  required 
under  PBA?  It  cannot.  Therefore,  a  requirement  that  meets  the  conditions  for  LOE  does 
not  meet  the  conditions  for  PBA. 

The  contrapositive  of  this  statement  is  also  true:  A  requirement  that  meets  the 
conditions  for  PBA  does  not  meet  the  conditions  for  LOE.  If  the  desired  performance 
outcomes  are  known  (as  they  must  be  under  a  PBA),  the  requirement  can  therefore  be 
clearly  defined,  and  estimates  of  extent,  duration,  and  costs  can  logically  be  developed. 
Therefore,  when  PBA  is  appropriate,  the  conditions  for  using  LOE  are  not  present. 

As  demonstrated  by  the  examination  of  PBA  and  LOE  from  a  rational 
perspective,  the  dichotomous  relationship  between  the  two  approaches  makes  sense. 
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3.  The  Dichotomous  Relationship  is  Apparent  in  the  Data 

Question  #lc  of  the  research  is:  Is  the  dichotomous  relationship  between  PBA  and 
LOE  apparent  in  the  data?  The  dichotomous  relationship  between  PBA  and  LOE  does 
not  exist  only  in  regulations,  policy,  and  theory;  it  was  apparent  in  the  data.  If  the 
Councils  were  correct  in  the  belief  that  PBA  can  be  applied  to  LOE,  one  would  expect  to 
find  at  least  one  such  instance  in  the  sample;  yet  not  a  single  LOE  contract  in  the  sample 
had  properly  applied  PBA.  The  contracts  most  commonly  failed  by  stating  a  number  of 
hours  to  be  provided  rather  than  specifying  a  clear  performance  result  (95%).  It  should  be 
noted  that  only  one  LOE  contract  specified  a  clear  perfonnance  result  and  also  specified 
a  number  of  hours.8  This  contract,  N00173-14-D-2004/0005,  was  for  research  and 
development  and  specified  the  delivery  of  a  prototype  as  the  end  result.  This  contract 
failed  to  meet  the  minimum  PBA  criteria  because  it  specified  both  hours  and  a  result;  as 
described  in  FAR  37.602(b),  the  contract  must  specify  either  one  or  the  other,  but  not 
both.  Therefore,  no  contract  in  the  sample  properly  applied  PBA  because  no  contract  met 
the  single  criterion  listed  at  FAR  37.602(b)(1). 

Observations  from  the  research  illustrate  how  the  sampled  LOE  contracts  failed  to 
meet  this  criterion.  In  many  cases,  the  lack  of  clear  performance  results  appeared  to  be 
due  to  the  complexity  of  the  requirements.  For  example,  several  contracts  focused  only 
on  the  delivery  of  technical  data  such  as  engineering  studies,  analyses,  investigations, 
technical  reports,  test  plans,  and  program  management  documentation,  without  tying 
these  to  a  clear  outcome.  Additionally,  the  researcher  observed  that  many  contracts 
required  deliverables  on  an  ad  hoc  rather  than  a  finite  basis.  However,  it  is  important  to 
note  that  the  lack  of  perfonnance  results  does  not  necessarily  reflect  a  failure  on  behalf  of 
the  government.  Quite  often,  the  government  cannot  define  clear  outcomes  or  even  a 
finite  set  of  deliverables  prior  to  the  award  of  a  contract. 

OUSD(AT&L)  recognizes  this  when  it  states  that  in  some  cases,  the  government 
“cannot  finnly  predict  the  tasks,  efforts,  and  required  outcomes”  (OUSD(AT&L),  2014, 

8  The  contract  apparently  ignored  the  guidance  at  FAR  35.005(c),  which  states,  “...contracting  officers 
should  ensure  that  language  suitable  for  a  level-of-effort  approach. .  .is  not  intermingled  with  language 
suitable  for  a  task-completion  approach”  (FAR,  2015). 
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p.  2).  This  is  also  recognized  in  an  article  entitled,  “A  Proposal  for  a  New  Approach  to 
Performance-Based  Services  Acquisition,”  by  Vernon  J.  Edwards  and  Ralph  C.  Nash. 
The  authors  suggest  that  for  long-tenn  and  complex  services  requirements: 

It  is  unrealistic  to  ask  agencies  to  specify  services  at  the  time  of  contract 
award  in  clear,  specific,  objective,  and  measureable  terms  when  future 
needs  are  not  fully  known  or  understood.  (Edwards  &  Nash,  Jr.,  2007, 
p.  355) 

Regarding  the  provision  of  deliverables  on  an  ad  hoc  basis,  Edwards  and  Nash 
note  that 

parties  to  long-term  and  complex  service  contracts... engage  in  ad  hoc 
decision  making  in  response  to  emerging  and  changing  requirements, 
shifting  priorities,  and  unexpected  circumstances.  (Edwards  &  Nash,  Jr., 

2007,  p.  355) 

The  complex  nature  of  many  sampled  contracts  helps  explain  why  the  LOE 
contract  type  was  selected,  why  performance  results  were  not  clearly  defined,  and  thus, 
why  effective  application  of  PBA  was  not  possible. 

4.  Question  #1  Conclusion 

The  analysis  of  Question  #1  demonstrated  that  regulations  and  recent  policy 
establish  a  dichotomous  relationship  between  PBA  and  LOE.  Contracting  officers  must 
choose  either  one  of  these  approaches  or  the  other,  but  not  both.  Furthennore,  upon 
examining  the  underlying  theories  of  PBA  and  LOE,  it  is  clear  that  this  dichotomous 
relationship  is  rational.  The  two  approaches  are  fundamentally  incompatible.  Given  that 
regulations  and  policy  make  such  a  rational  distinction,  and  given  the  complex  nature  of 
the  contracts  sampled,  it  is  not  surprising  to  discover  that  the  Navy  did  not  appropriately 
apply  PBA  techniques  to  LOE  contracts.  However,  it  was  surprising  to  discover  how 
often  the  Navy  attempts  to  combine  these  two  conflicting  approaches.  This  leads  to 
Question  #2. 

C.  POSSIBLE  CAUSES  FOR  COMBINING  PBA  AND  LOE 

Question  #2  of  the  research  is:  Why  does  the  Navy  attempt  to  use  PBA  techniques 
for  LOE  contracts?  Although  PBA  and  LOE  are  two  distinct  approaches  that  should  not 
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be  used  together,  the  research  shows  that  the  Navy  often  combines  these  concepts.  Of  the 
50  contracts  sampled,  19  contracts  (38%)  were  not  clearly  identified  as  LOE  or 
completion.9  Excluding  the  “not  clear”  category,  22  of  the  remaining  3 1  contracts  were 
identified  as  LOE  (71%).  This  is  depicted  in  Figure  9. 


Completion,  9, 
29% 


Total  LOE,  22, 
71% 


Figure  9.  LOE  vs.  Completion,  Excluding  “Not  Clear” 

Excluding  those  that  are  not  clear,  we  can  reasonably  assume  that  a  substantial 
percentage  of  the  Navy’s  T&M,  LH,  FFP-LOE  and  cost-reimbursement  contracts  that  are 
reported  as  PBA  have  inappropriately  applied  PBA.10  What  is  motivating  the  Navy  to 
combine  these  approaches? 

The  analysis  of  this  question  is  divided  into  three  parts.  This  question  requires  an 
examination  of  the  history  of  the  PBA  policy  initiative,  the  incentives  it  creates  for 
contracting  activities,  and  the  extent  to  which  services  acquisition  literature  specifically 
considers  the  LOE  contract  type.  The  analysis  reveals  that  PBA  was  over-applied. 


9  This  is  likely  a  separate  problem  in  and  of  itself.  Although  lack  of  access  to  the  specification  is  not 
necessarily  a  problem,  one  could  argue  that  the  contract  should  state  the  specific  contract  type  (i.e.,  LOE  or 
completion)  clearly,  and  independent  of  any  contract  attachments.  The  intermingling  of  contract  types  is 
likely  more  problematic.  However,  this  potential  problem  is  outside  of  the  scope  of  this  study. 

10  Even  after  applying  a  large  potential  margin  of  error,  this  magnitude  is  significant. 
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1.  History  of  the  PBA  Policy  Initiative 

Question  #2a  of  the  research  is:  Does  the  history  of  the  PBA  policy  initiative 
explain  why  the  Navy  combines  PBA  and  LOE?  A  review  of  the  PBA-LOE  conflict 
within  its  historical  context  is  necessary  to  explore  why  the  Navy  is  combining  the  two 
approaches.  History  suggests  that  the  contracting  community  was  pushed  to  implement  a 
new  popular  policy,  and  thus  this  policy  was  over-applied. 

The  history  of  PBA  is  well-summarized  in  a  thesis  by  Christos  Avramidis,  “An 
Analysis  of  Performance-Based  Service  Acquisition  and  its  Applicability  to  Hellenic 
Navy  Service  Acquisition  Activities.”  Avramidis’  summary  begins  in  1980,  when  the 
Office  of  Federal  Procurement  Policy  (OFPP)  issued  OFPP  Pamphlet  Number  4,  “A 
Guide  for  Writing  and  Administering  Performance  Statements  of  Work  for  Service 
Contracts”  (Avramidis,  2012,  p.  7-8). 11  However,  as  Avramidis’  summary  captures,  PBA 
transformed  into  a  movement  throughout  the  1990s  and  into  the  mid-2000s.  The  PBA 
concept  grew  from  being  a  policy  preference  in  1991  to  a  statutory  preference  carrying 
mandatory  reporting  requirements  and  implementation  goals.  Several  agencies  created 
guidebooks  on  using  PBA,  including  OFPP  and  DOD.  Even  an  Interagency-Industry 
Partnership  created  guidance  (Avramidis,  2012,  p.  9).  A  brief  review  of  this  history 
demonstrates  that  the  PBA  concept  was  quite  popular  among  both  agency  and 
congressional  policy-makers. 

The  popularity  of  PBA  is  best  reflected  in  the  growth  of  DOD’s  PBA 
implementation  goals.  In  2001,  the  Office  of  Management  and  Budget  (OMB)  created  a 
goal  of  using  PBA  for  20%  of  total  eligible  services  dollars  obligated.  OMB  raised  this 
goal  to  40%  in  2004,  to  45%  in  2006,  then  ultimately  to  50%  in  2008  (Avramidis,  2012, 
p.  59-61). 

The  PBA  policy  initiative,  including  the  implementation  goals,  certainly  had  an 
impact.  Contracts  identified  as  having  used  PBA  techniques  increased  substantially  from 


1 1  In  “A  Proposal  for  a  New  Approach  to  Performance-Based  Services  Acquisition,”  Edwards  and 
Nash  note  several  attempts  to  use  PBA  prior  to  1980  (Edwards  &  Nash,  Jr.,  2007,  p.  354). 
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fiscal  year  2005  to  2014.  In  2005,  DOD  reported  using  PBA  techniques  for  29%  of  total 
obligations  for  services;  in  2014,  this  had  increased  to  57%. 12 

However,  as  noted  in  a  2002  Government  Accountability  Office  (GAO)  Report 
entitled,  “Guidance  Needed  for  Using  Performance-Based  Service  Contracting,”  several 
agencies  that  claimed  to  have  applied  PBA  did  not  actually  do  so.  Over  half  of  the 
contracts  sampled  by  GAO  (16  of  25)  did  not  actually  meet  the  minimum  criteria  for 
PBA  (Government  Accountability  Office  [GAO],  2002,  p.  2). 

The  fact  that  the  growth  in  PBA  goals  is  accompanied  with  a  growth  in  the 
percentage  of  PBA  dollars  obligated  suggests  that  the  PBA  push  was  strong — the 
acquisition  community  clearly  responded  to  this  pressure.  However,  the  fact  that  this 
growth  is  accompanied  with  pseudo-PBA  contracts,  as  observed  in  GAO’s  report  and  in 
this  study,  suggests  that  the  acquisition  community’s  response  was  not  what  the  policy¬ 
makers  had  in  mind.  Agencies,  including  the  Navy,  seem  compelled  to  identify  contracts 
as  PBA  whether  or  not  this  approach  is  a  proper  fit.  The  negative  incentives  placed  on 
contracting  activities  are  explored  further  by  the  following  question. 

2.  Misguided  Incentives  for  Contracting  Activities 

Question  #2b  of  the  research  is:  What  incentives  does  the  PBA  policy  initiative 
give  contracting  activities?  Aside  from  the  preferential  treatment  given  to  PBA  in  the 
regulations,  the  PBA  policy  initiative  motivated  the  contracting  community  to  adopt  PBA 
techniques  in  two  key  ways:  1)  goals  and  reporting,  and  2)  required  waivers  for  non-PBA 
contracts.  Taken  together,  these  strongly  incentivize  contracting  commands  to  identify 
contracts  as  PBA.  These  incentives  are  apparently  so  strong  that  the  motivation  to 
identify  a  services  contract  as  PBA  overrides  the  motivation  to  apply  PBA  techniques 
only  when  they  are  proper. 

The  effect  of  goals  and  reporting  on  the  misapplication  of  PBA  to  LOE  may  be 
partially  explained  by  how  reporting  figures  are  calculated.  A  memorandum  entitled, 
“Performance-Based  Services  Acquisition  (PBSA),”  issued  by  OUSD(AT&L)  on  1 


1 2  According  to  data  from  FPDS. 
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February  2006,  provides  updated  exemptions  from  PBA  reporting  for  certain  types  of 
services  (OUSD(AT&L),  2006).  However,  these  exemptions  have  neglected  many  types 
of  services  where  PBA  is  not  appropriate. 

Edwards  and  Nash  contend  that  PBA  has  been  over-applied.  They  argue,  quite 
effectively,  that  traditional  PBA  is  “not  a  practical  approach  to  buying  long-tenn  and 
complex  services”  (Edwards  &  Nash,  Jr.,  2007,  p.  355).  However,  the  exemptions  from 
PBA  reporting  do  not  completely  cover  long-tenn  and  complex  services,  which  may 
reasonably  be  appropriate  for  the  LOE  approach.  Because  LOE  services  contracts  are  not 
exempted  from  PBA  reporting,  activities  may  identify  LOE  contracts  as  PBA  in  order  to 
meet  PBA  goals. 

The  second  motivation,  requiring  higher  level  approvals  for  non-PBA  contracts, 
may  have  also  contributed  to  the  misapplication  of  PBA.  The  National  Defense 
Authorization  Act  for  Fiscal  Year  2002  required  higher  level  approvals  of  contracts  that 
were  not  performance-based.  This  rule  was  implemented  in  DOD  as  an  interim  rule  in 
Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  237.170-2(a)  and  DFARS 

237.170- 3(a)  (68  FR  56563).  This  requirement  was  later  consolidated  into  DFARS 

237.170- 2(a)  (70  FR  29643).  Although  Congress  later  abandoned  the  higher  level 
approval  requirement  in  favor  or  more  general  urgings  to  use  PBA  (i.e.,  “to  the  maximum 
extent  practicable”)  the  requirement  was  never  removed  from  the  DFARS  (National 
Defense  Authorization  Act  for  Fiscal  Year  2006). 

DFARS  237.170-2(a)  provides  flexibility  to  the  agency  to  determine  the  specific 
approving  official.  Prior  to  19  March  2015,  the  Navy  had  assigned  the  Head  of  the 
Contracting  Agency  (HCA)  or  the  Deputy  Assistant  Secretary  of  the  Navy  for 
Acquisition  and  Procurement  (DASN(AP))  as  the  approving  official,  depending  on  the 
dollar  value  of  the  contract  (NMCARS  5237.170-2(a)  through  Change  13-04). 13  Per 
NMCARS  5237.170-2(a)(S-90),  only  architect-engineer  (A&E)  services  and  personal 
medical  services  were  exempt  from  the  approval  requirements.  Therefore,  a  Navy 

13  The  Navy  recently  nullified  this  approval  requirement  via  NMCARS  Change  13-05,  by  designating 
the  contracting  officer  as  the  approval  authority.  It  has  yet  to  be  seen  what  kind  of  impact  this  will  have  on 
the  use  of  PBA. 
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contracting  officer  faced  with  a  complex  services  requirement  would  have  needed  at  least 
the  approval  of  the  HCA  before  awarding  a  contract  that  did  not  use  PBA  techniques. 

The  consequences  of  this  higher  level  approval  requirement  may  have  been 
harmful.  In  the  Navy,  the  HCA  is  typically  at  least  two  levels  higher  than  the  contracting 
officer  in  the  chain-of-command.  Contracting  officers  who  bought  complex  services, 
such  as  those  acquired  through  LOE  contracts,  were  faced  with  a  difficult  dilemma — 
either  attempt  to  mask  services  requirements  as  PBA  and  report  them  as  such,  or  submit  a 
waiver  to  the  HCA  for  every  complex  services  contract.  In  reality,  obtaining  HCA 
approval  for  every  contract  is  not  a  practical  approach.  As  demonstrated  by  the  data, 
many  contracting  officers  appear  to  have  chosen  to  either  misapply  PBA  or  misidentify 
contracts  as  PBA. 

The  strong  encouragement  to  use  PBA,  which  included  the  use  of  implementation 
goals,  reporting,  and  higher  level  approvals,  has  motivated  contracting  officers  to  over¬ 
apply  PBA  techniques.  This  helps  explain  why  every  LOE  contract  sampled  had 
ineffectively  combined  the  PBA  and  LOE  approaches. 

Although  incentives  have  clearly  had  an  impact,  this  researcher  (also  a  contracting 
officer)  recognizes  that  some  contracting  officers  may  be  confused  about  the  relationship 
between  PBA  and  LOE.  In  other  words,  some  contracting  officers  are  not  consciously 
choosing  either  to  pursue  their  HCA’s  approval  or  to  mask  an  LOE  requirement  as  PBA; 
they  may  never  have  learned  that  the  two  approaches  are  incompatible.  This  leads  to  the 
next  question. 


3.  Gaps  in  Services  Acquisition  Literature 

Question  #2c  of  the  research  is:  Does  literature  on  PBA  make  clear  that  it  does 
not  apply  to  LOE?  Despite  the  distinction  in  the  FAR,  and  despite  volumes  of  guidance 
on  how  to  use  PBA,  the  researcher  found  a  surprising  lack  of  discussion  on  PBA’s 
application  to  LOE  contracts.  This  apparent  gap  in  the  literature  may  indicate  that 
contracting  officers,  or  perhaps  even  some  acquisition  leaders,  are  confused  about  the 
relationship  between  PBA  and  LOE. 
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The  DOD  Guidebook  for  the  Acquisition  of  Services  does  not  specifically  address 
the  relationship  between  PBA  and  LOE.  Although  the  Guidebook  acknowledges  that 
PBA  is  not  appropriate  for  all  requirements,  it  does  not  specifically  cite  services 
requirements  that  are  fit  for  the  LOE  contract  type  (DOD,  2012,  p.  9).  Rather,  the 
Guidebook  seems  to  suggest  that  even  complex  services  requirements  can  be  re¬ 
structured  to  fit  the  PBA  model.  For  example,  the  Guidebook  advises  that 

you  need  to  keep  the  focus  on  what  mission  outcomes  you  are  trying  to 
achieve. .  .If  you  can  keep  a  higher  view  of  what  you’re  asking  a  contractor 
to  accomplish,  you  will  have  far  more  success  in  implementing  a 
perfonnance-based  approach  for  your  service  requirements.  (DOD,  2012, 

P-10) 

The  Guidebook  identifies  basic  task  statements  such  as  “Conduct  a  study  on...” 
and  “Review  and  assess...”  and  suggests  that  these  can  be  appropriate  for  PBA  (DOD, 
2012,  p.  26).  However,  these  statements  could  reasonably  indicate  a  complex  services 
requirement,  thus  potentially  fit  for  LOE.  The  Guidebook  does  not  acknowledge  that  if 
such  services  are  appropriate  for  LOE  type  contracts,  PBA  should  not  be  used. 
Therefore,  the  Guidebook  seems  to  ignore  that  FAR  37.602(b)  and  recent  policy  create  a 
distinct  choice  between  PBA  and  LOE. 

Notably,  the  Guidebook’s  real  life  examples  of  how  to  convert  a  non-PBA 
requirement  to  PBA,  specifically  the  dredging  of  a  river  and  taxi  services,  are 
noncomplex  (DOD,  2012,  p.  10  and  p.  27).  The  Guidebook  does  not  attempt  such  a 
venture  for  program  management  support  services,  likely  because  the  complexity  of  such 
a  requirement  would  not  lend  itself  to  the  PBA  approach. 

Similar  shortcomings  are  seen  in  OFPP’s  A  Guide  to  Best  Practices  for 
Performance-Based  Service  Contracting.  In  fact,  this  document  fails  much  more  directly 
in  recognizing  that  PBA  and  LOE  are  distinct  choices.  It  states,  “when  the  use  of  time 
and  materiaFlabor  hour  contracts  is  appropriate,  agencies  should  employ  PBSC  methods 
to  the  maximum  extent  feasible”  (Office  of  Federal  Procurement  Policy  [OFPP],  1998, 
Chapter  6). 14  Oddly,  this  guide  also  recognizes  that  “application  of  only  selected  aspects 


14  Performance-Based  Service  Contracts  (PBSC)  is  another  name  for  PBSA. 
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of  the  total  PBSC  methodology  is  not  likely  to  be  successful”;  this  negates  the  Guide’s 
aforementioned  suggestion  that  PBA  techniques  could  be  applied  on  a  partial  basis 
(OFPP,  1998,  Chapter  2). 

Perhaps  the  most  comprehensive  instruction  on  PBA  is  a  course  developed  by 
National  Contract  Management  Association  (NCMA)  entitled  “Performance-Based 
Service  Acquisition — Fonning  Performance-Based  Contracts  for  Acquiring  Services.” 
This  288-page  course  text  also  acknowledges  that  “PBSA  is  not  a  ‘one  size  fits  all’ 
process,”  yet  it  fails  to  discuss  PBA’s  applicability  to  LOE  (Corporate  Learning 
Solutions,  2006,  p.  2-xiii). 

Despite  observing  a  general  lack  of  discussion  on  the  PBA-LOE  relationship,  the 
researcher  was  able  to  find  two  sources  that  addressed  the  conflict.  However,  unlike  the 
DOD  Guidebook  for  the  Acquisition  of  Services  and  OFPP’s  “A  Guide  to  Best  Practices 
for  Performance-Based  Service  Contracting,”  it  is  unlikely  that  these  more  obscure 
sources  would  be  accessed  by  a  typical  contracting  officer  in  search  of  guidance.  The  first 
source,  71  FR  211,  was  discussed  earlier  in  this  Chapter;  this  public  comment  and 
response  from  nearly  a  decade  ago  is  not  considered  to  be  reliable  guidance  for  a 
working-level  contracting  officer.  The  second  source,  an  OFPP  report  from  July  2003, 
“Performance-Based  Service  Acquisition:  Contracting  for  the  Future,”  concluded  that 
PBA  and  LOE  naturally  conflict.  This  report  recommends,  rather  than  implements, 
changes  to  regulations  and  guidance;  it  is  also  not  considered  to  be  reliable  guidance. 

Based  on  this  review,  there  is  an  apparent  lack  of  discussion  regarding  the  PBA- 
LOE  relationship  in  common  services  acquisition  literature.  The  fact  that  PBA  and  LOE 
are  two  distinct  choices  in  the  regulations  is  often  overlooked.  Given  that  LOE  is  a  valid 
contract  type,  including  cost-reimbursement  term,  FFP-LOE,  T&M,  and  LH  contracts, 
this  is  quite  surprising.  Even  a  studious  contracting  officer,  one  who  reviewed  his/her 
Command’s  guidance,  DOD’s  guidance,  OFPP’s  guidance,  and  attended  a  full-length 
master’s  level  course  on  PBA  may  not  understand  that  PBA  and  LOE  should  not  be  used 
together. 
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4.  Question  #2  Conclusion 

The  analysis  of  Question  #2  demonstrated  that  PBA  was  heavily  encouraged  by 
DOD  acquisition  leaders,  and  this  encouragement  caused  an  over-application  of  PBA  to 
the  LOE  contract  type.  The  PBA  policy  initiative  included  perverse  incentives  that 
motivated  the  Navy  to  apply  PBA  techniques  to  LOE  contracts  inappropriately. 
Furthennore,  an  analysis  of  the  services  acquisition  literature  revealed  a  surprising  lack 
of  discussion  on  the  applicability  of  PBA  to  LOE  contracts.  Some  contracting  officers 
may  be  confused  about  the  PBA-LOE  relationship.  To  such  a  contracting  officer,  the 
combination  of  PBA  and  LOE  may  seem  like  a  correct  approach.  So  why  should 
contracting  officers  avoid  combining  these  two  concepts?  This  leads  to  Question  #3. 

D.  CONSEQUENCES  OF  COMBINING  PBA  AND  LOE 

Question  #3  of  the  research  is:  What  are  the  consequences  of  using  PBA 
techniques  for  LOE  contracts?  Some  may  argue  that  although  PBA  techniques  are  not 
entirely  suitable  for  LOE  services,  the  techniques  are  useful  and  should  still  be  applied. 
Perhaps  the  process  of  thinking  about  perfonnance  results,  even  though  they  may  not  be 
in  clear,  specific,  and  objective  tenns,  forces  requirements  personnel  to  think  about 
possible  outcomes  and  at  least  improves  the  quality  of  the  specification.  In  other  words,  a 
proponent  of  this  argument  would  suggest  that  it  cannot  hurt  to  make  requirements 
personnel  specify  outcomes.  The  DOD  Acquisition  of  Services  Guidebook,  the  OFPP 
Guide,  and  the  NCMA-developed  course  text  suggest  that  pressing  the  PBA  approach  can 
result  in  transforming  non-PBA  requirements  to  PBA  requirements. 

However,  this  argument  ignores  the  negative  consequences  of  using  PBA  for 
LOE;  these  potential  negative  consequences  must  be  explored  and  considered.  The 
analysis  of  this  question  is  divided  into  three  parts.  First,  the  analysis  addresses  possible 
perfonnance  incentives.  Second,  the  analysis  considers  possible  administrative  burden. 
Finally,  the  analysis  evaluates  the  notion  that  applying  PBA  to  LOE  is  beneficial.  The 
analysis  ultimately  demonstrates  that  combining  PBA  and  LOE  can  result  in  misguided 
perfonnance  incentives,  confusion,  and  unnecessary  administrative  burden. 
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1.  Useless  and  Distracting  Performance  Incentives 

Question  #3a  of  the  research  is:  What  kind  of  perfonnance  incentives  are  created 
when  PBA  is  applied  to  LOE,  and  what  impacts  do  these  incentives  have?  An 
examination  of  the  sampled  contracts  revealed  that  applying  PBA  to  LOE  resulted  in 
useless  and  distracting  performance  incentives. 

The  researcher  observed  several  perfonnance  incentives  that  were  obvious  or 
redundant.  For  example,  several  sampled  contracts  contained  a  perfonnance  standard 
similar  to  the  following:  “100%  of  reports  are  timely,  accurate,  and  complete.”  This 
standard  should  be  obvious  to  any  services  contractor;  it  is  hard  to  imagine  a  contractor 
altering  its  perfonnance  based  on  this  information.  Furthermore,  monitoring  performance 
and  providing  feedback  to  the  contractor  in  these  areas  is  redundant  for  any  contract  rated 
in  the  Contractor  Performance  Assessment  Rating  System  (CPARS).15  CPARS  rating 
areas  already  include  Quality  of  Product  or  Service,  Schedule,  and  Cost  Control 
(Contractor  Performance  Assessment  Rating  System  [CPARS],  2014).  Therefore, 
specifying  that  reports  must  be  timely,  accurate,  and  complete  is  redundant.16 

On  the  other  end  of  the  spectrum,  many  sampled  LOE  contracts  contained 
standards  that  created  distracting  incentives.  Some  standards  focused  the  contractor’s 
attention  on  trivial  elements  of  the  overall  performance,  which  may  have  distracted  the 
contractor’s  attention  away  from  important  elements.  For  example,  several  contracts 
focused  on  grammatical  correctness  of  reports.  Describing  a  complex  services 
requirement  in  terms  of  the  number  of  grammatical  errors  ignores  the  inherent 
subjectivity  in  evaluating  complex  performance  outcomes.  Therefore,  a  contractor 
providing  these  services  may  have  been  incentivized  to  provide  grammatically  correct 
reports  (which  can  be  objectively  measured),  rather  than,  for  example,  an  innovative 
solution  to  a  complex  engineering  problem  (which  can  only  be  subjectively  measured). 


15  Applicability  of  CPARS  reporting  is  based  on  dollar  value  of  the  contract  and  the  business  sector. 
Business  sectors  include  Professional/Technical  &  Management  Support  Services  and  Information 
Technology  Services,  where  services  requirements  are  typically  complex  in  nature  and  could  reasonably  fit 
the  LOE  contract  type  (CPARS,  July  2014,  p.  Al-3). 

16Although  CPARS  affects  only  future  contracts,  it  provides  incentives  for  contractors  to  perform 
under  active  contracts. 
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Drawing  attention  towards  a  meaningless  objective  may  distract  attention  away  from  a 
mission-focused  objective,  thus  resulting  in  harm  to  the  government’s  mission. 

Elliott  Branch,  DASN(AP),  has  advised  contracting  officers  to  focus  on  the 
business  deal,  specifically  what  the  government  is  really  buying  and  what  the 
government  is  paying  (Burke,  2014).  The  results  of  applying  PBA  to  LOE  can  pollute  the 
contracting  officer’s  understanding  of  the  business  deal.  Surely,  the  government  is  not 
really  buying  grammatically  correct  reports.  After  all,  a  grammatical  report  is  useless  if  it 
adds  no  value  to  the  agency’s  mission.  Under  an  LOE  contract,  the  government  is  really 
buying  a  service  to  be  performed  at  a  specified  level  of  effort. 

As  observed  in  the  sampled  contracts,  when  the  government  cannot  define 
outcomes  in  clear,  specific,  and  objective  terms  with  measureable  outcomes,  forced 
attempts  to  do  so  are  not  effective.  This  research  suggests  that  attempts  to  combine  PBA 
and  LOE,  two  naturally  incompatible  approaches,  can  result  in  useless  and  distracting 
perfonnance  incentives,  which  may  hann  the  government’s  true  objectives. 

2.  Unnecessary  Administrative  Burden 

Question  #3b  of  the  research  is:  What  administrative  burden  is  involved  with 
mixing  PBA  and  LOE?  Forcing  the  PBA  approach  on  a  complex  services  requirement 
requires  substantial  effort  during  both  the  procurement  and  administration  phases.  This 
unnecessary  administrative  burden  is  another  negative  effect  of  combining  PBA  and 
LOE. 

Prior  to  issuing  the  solicitation,  the  government  team  must  define  the  requirement. 
It  is  both  awkward  and  time-consuming  to  create  a  PWS  and  QASP  tied  to  objectively 
measurable  outcomes  for  a  requirement  that  cannot  be  defined  in  clear,  specific,  and 
objective  terms  with  measureable  outcomes.  The  contracting  officer,  often  charged  with 
advocating  the  PBA  approach,  may  ask  requirements  personnel  difficult  questions  in 
hopes  of  arriving  at  performance  results  that  can  be  objectively  measured.  For  example, 
the  DOD  Guidebook  for  the  Acquisition  of  Services  encourages  the  acquisition  team  to 
ask  “Why?,”  until  some  objectively  measureable  criteria  are  apparent  (DOD,  2012, 
p.  26).  Furthermore,  requirements  personnel  are  encouraged  to  develop  a  QASP  that 
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defines  the  required  objectives,  standards,  acceptable  quality  levels,  inspection  methods, 
and  incentives  (DOD,  2012,  p.  34-35). 

Because  requirements  that  fit  the  LOE  contract  type  do  not  have  clear,  specific, 
and  objective  terms  with  measureable  outcomes,  this  process  has  the  potential  to 
consume  valuable  time  and  create  tension  between  the  contracting  officer  and 
requirements  personnel.  The  time  and  effort  spent  forcing  the  PBA  approach  on  LOE 
services  prior  to  contract  award  represents  a  cost  to  the  government. 

Furthennore,  the  standards  and  acceptable  quality  levels,  although  meaningless  or 
distracting,  must  be  considered  by  the  contractor  in  developing  the  proposal.  This  time 
and  effort  wasted  prior  to  award  represents  a  cost  to  private  industry. 

There  is  additional  burden  after  contract  award,  during  the  administration  phase. 
The  contractor  must  perform  to  the  required  standards  and  acceptable  quality  levels,  and 
the  government  must  monitor  this  performance  in  accordance  with  the  QASP.  One 
sampled  contract  required  that  technical  reports  contain  grammatical  errors  in  no  more 
than  2%  of  the  lines.  The  contractor  must  therefore  review  every  line  of  these  reports 
prior  to  submission  to  ensure  that  they  contain  no  grammatical  errors.  Upon  receipt  of  the 
reports,  the  government  must  also  review  every  line,  count  the  grammatical  errors,  and 
then  record  the  number  of  reports  meeting  this  acceptable  quality  level.  It  was  described 
earlier  that  a  focus  on  grammatical  correctness  is  a  distraction  from  the  true  requirement; 
it  is  also  a  waste  of  time  and  effort. 

As  demonstrated  in  this  analysis,  forcing  the  PBA  approach  on  a  complex 
services  requirement  has  potential  to  require  substantial  effort  during  both  the 
procurement  and  administration  phases.  This  cost  to  both  the  government  and  industry  is 
another  negative  consequence  of  combining  PBA  and  LOE. 

3.  PBA  and  LOE  can  be  Separated 

Question  #3c  of  the  research  is:  What  happens  when  PBA  is  not  used  with  LOE? 
The  two  previous  questions  demonstrated  that  using  PBA  and  LOE  together  carries 
negative  consequences,  in  the  form  of  meaningless  or  distracting  incentives  and  burden 
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during  the  procurement  and  administrative  phases.  This  question  analyzes  what  would 
happen  if,  as  the  regulations  direct,  the  PBA  approach  was  no  longer  applied  to  LOE. 

Critics  of  this  approach  may  argue  that  in  the  absence  of  PBA,  requirements 
personnel  would  not  be  not  challenged  to  define  performance  outcomes,  and 
opportunities  for  proper  use  of  PBA  could  be  missed.  Furthermore,  critics  may  argue  that 
without  a  fonnal  QASP  tied  to  objectively  measurable  outcomes,  government  oversight 
of  contractor  perfonnance  would  suffer.  These  critics  may  conclude  that  the  over¬ 
application  of  PBA  is  necessary.  This  analysis  explores  this  notion  and  ultimately 
challenges  it. 

First,  critics  of  separating  PBA  and  LOE  may  argue  that  the  PBA  ideology 
challenges  requirements  personnel  to  define  outcomes,  thus  identifying  opportunities  for 
the  proper  use  of  PBA.  The  identification  of  such  an  opportunity  is  discussed  in  the  DOD 
Guidebook  for  the  Acquisition  of  Services:  “After  some  prolonged  and  heated  discussion, 
they  [the  Corps  of  Engineers]  detennined  that... keeping  the  channel  open  was  their 
perfonnance  objective,  not  dredging”  (DOD,  2012,  p.  10). 

Surely,  some  requirements  are,  by  their  nature,  more  appropriate  for  PBA  than 
LOE.  In  that  sense,  some  requirements  originally  described  as  LOE  may  be  re-structured 
to  be  completion,  thus  allowing  the  application  of  PBA  techniques. 

However,  the  over-application  of  PBA  to  LOE  is  not  a  responsible  or  necessary 
solution.  This  approach  is  not  responsible  because,  as  observed  in  the  research  data,  the 
Navy  has  adopted  this  approach  ineffectively;  complex  services  that  are  truly  appropriate 
for  LOE  may  suffer  from  the  over-application  of  PBA.  Furthennore,  this  over-application 
is  not  necessary.  The  contracting  officer  must  choose  a  contract  type  in  accordance  with 
the  FAR.  FAR  Part  16  provides  rules  for  the  application  of  each  contract  type,  and  these 
rules  include  a  preference  for  the  completion  type  over  the  LOE  type.  Therefore,  if 
requirements  personnel  submit  a  requirement  for  a  specified  number  of  hours,  but  it  can 
clearly  be  converted  to  a  completion  type  requirement  with  further  definition,  the 
contracting  officer  must  work  with  requirements  personnel  to  do  so.  In  other  words, 
separating  PBA  and  LOE  does  not  eliminate  the  contracting  officer’s  duty  to  select  the 
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appropriate  contract  type.  If  the  established  rules  are  followed,  and  if  an  LOE  contract 
type  is  selected,  PBA  should  not  be  considered — the  two  are  incompatible  by  definition. 
The  over-application  of  PBA  is  not  necessary  to  motivate  contracting  officers  to  comply 
with  FAR  37.602(a)  or  the  application  and  limitation  rules  of  FAR  Part  16. 

Second,  critics  of  separating  PBA  and  FOE  may  argue  that  government  oversight 
of  contractor  performance  would  suffer  without  a  formal  QASP  tied  to  objectively 
measurable  outcomes.  Indeed,  the  government  must  have  a  plan  for  monitoring 
contractor  performance.  However,  requiring  a  fonnal  QASP  that  is  tied  to  objectively 
measureable  outcomes  is  clearly  not  an  effective  solution  for  FOE  services.  As  seen  in 
the  sampled  contracts,  QASPs  for  FOE  contracts  focused  on  obvious  or  distracting 
outcomes,  rendering  them  ineffective. 

The  researcher  notes  that  an  alternate  surveillance  plan,  one  that  recognizes  the 
inherent  subjectivity  of  monitoring  the  performance  of  complex  services,  could  be 
developed.  Such  a  surveillance  plan  could  mirror  the  past  performance  assessment 
criteria  of  CPARS.17  During  CPARS  evaluations,  contractors  can  be  judged  on  quality, 
schedule,  and  cost  control  without  using  a  fonnal  QASP  tied  to  objectively  measureable 
outcomes.  Such  a  plan  could  be  standardized,  eliminating  the  administrative  burden 
associated  with  tailoring  a  plan  to  a  specific  requirement.  In  fact,  the  Space  and  Naval 
Warfare  Systems  Center  Pacific  (SSC  Pacific)  has  adopted  such  a  plan  for  all  non-PBSA 
contracts.  The  Non-PBSA  Surveillance  Plan  is  attached  for  reference  as  Appendix  E.  The 
use  of  a  fonnal  QASP  tied  to  objectively  measureable  outcomes  is  not  necessary  to 
ensure  adequate  oversight  of  contractor  performance. 

While  some  may  argue  that  the  over-application  of  PBA  ultimately  results  in 
better  requirements  definition  and  government  oversight,  this  argument  ignores  that  such 
over-application  can  be  harmful  and  is  unnecessary  in  achieving  these  desired  goals.  PBA 
and  FOE  can  be  separated  without  sacrificing  the  performance-based  preference  at  FAR 
37.602(a)  or  proper  contract  oversight. 


17  CPARS  recognizes  subjective  evaluation  of  contractor  performance  with,  “Each  area  assessment 
must  be  supported  by  objective  data  (or  subjective  observations)...”  (CPARS,  July  2014,  p.  A3-8). 


38 


4.  Question  #3  Conclusion 

The  analysis  of  Question  #3  has  explored  the  negative  consequences  of 
combining  PBA  and  LOE.  Specifically,  combining  the  two  approaches  can  result  in 
meaningless  or  distracting  performance  incentives,  as  well  as  unnecessary  administrative 
burden.  Finally,  the  analysis  challenged  the  notion  that  PBA  must  be  over- applied  to 
ensure  proper  requirements  definition  and  government  oversight. 
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V.  CONCLUSION  AND  RECOMMENDATIONS 


A.  RESEARCH  FINDINGS 

This  study  has  explored  the  relationship  of  the  PBA  contracting  approach  to  LOE 
services.  Through  Question  #1,  it  was  discovered  that  PBA  techniques  cannot  be  used 
together  effectively,  because  there  is  a  natural  conflict  between  these  two  approaches.  It 
was  found  that  regulations  and  policy  create  a  distinct  choice  between  the  two 
approaches — either  PBA  or  LOE  can  be  used,  but  not  both.  By  examining  the  rationales 
behind  these  two  approaches,  it  was  found  that  this  dichotomous  relationship  is  logical. 
Despite  the  dichotomous  relationship,  a  review  of  50  services  contracts  revealed  that  the 
Navy  often  attempts  to  combine  the  two  approaches.  In  all  22  LOE  contracts  sampled, 
PBA  was  improperly  applied. 

Through  Question  #2,  this  study  suggests  that  there  are  three  main  reasons  why 
the  Navy  has  attempted  to  combine  PBA  and  LOE.  First,  the  history  suggests  that  amidst 
a  strong  push  by  acquisition  leadership  to  implement  PBA,  the  policy  was  over-applied  to 
LOE.  Second,  PBA  policy  included  implementation  goals,  reporting,  and  higher  level 
approval  requirements  that  motivated  the  Navy  to  apply  PBA  techniques  to  LOE 
contracts  inappropriately.  Third,  this  study  found  that  relevant  literature  does  not  make 
the  relationship  between  PBA  and  LOE  clear  ;  this  lack  of  clarity  further  contributed  to 
the  over-application  of  PBA  techniques  to  services  acquired  on  an  LOE  basis. 

Through  Question  #3,  this  study  found  that  combining  PBA  and  LOE  carries 
negative  consequences,  including  meaningless  or  distracting  incentives  and  unnecessary 
administrative  burden.  These  negative  consequences  represent  a  cost  to  both  government 
and  industry.  Finally,  this  study  challenged  the  notion  that  PBA  must  be  over- applied  to 
ensure  proper  requirements  definition  and  government  oversight.  The  conclusions  of 
this  study  inform  specific  recommendations  to  improve  the  acquisition  of  services  on  an 
LOE  basis. 
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B.  RECOMMENDATIONS 


This  section  offers  specific  recommendations  for  improving  the  acquisition  of 
services  on  an  LOE  basis.  The  recommendations  are  generally  presented  in  the  same 
order  as  the  analysis.  For  example,  recommendations  stemming  from  the  analysis  and 
conclusions  from  Question  #1  will  be  presented  first. 

1 .  Make  the  dichotomous  relationship  between  PBA  and  LOE  clear  in 
regulations  and  relevant  literature. 

•  Stress  in  the  FAR  that  PBA  methods  are  not  appropriate  for  LOE  contract 
types.  Add  the  following  exception  to  FAR  37.102(a)(1): 

(v)  Services  using  time-and-materials,  labor-hour,  cost-reimbursement 
term  form,  or  firm- fixed-price  level-of-effort  term  contract  types. 

This  was  originally  recommended  in  OFPP’s,  “Performance-Based 
Service  Acquisition:  Contracting  for  the  Future,”  yet  it  was  not  adopted  by 
the  Councils.  Although  FAR  37.602(b)(1)  creates  a  distinct  choice 
between  PBA  and  LOE,  LOE  is  not  explicitly  mentioned.  An  explicit 
exception  for  LOE  contracts  will  inform  contracting  officers  that  PBA  and 
LOE  cannot  be  combined. 

•  Represent  the  conflict  between  PBA  and  LOE  in  FPDS.  Establish  a 
reporting  field  in  FPDS  to  identify  LOE.  Automatically  populate  this  field 
as  “Yes”  for  T&M,  LH,  or  FFP-LOE  contract  types.  For  cost- 
reimbursement  types,  allow  the  contracting  officer  to  select  either  “Yes” 
or  “No.”  Link  this  reporting  field  to  the  PBSA  field.  If  “Yes  -  service 
where  PBA  is  used”  is  selected,  the  LOE  field  must  be  “No.”  If  the  LOE 
field  is  “Yes,”  PBSA  must  be  “No  -  service  where  PBA  is  not  used.”  In 
the  error  message,  direct  the  contracts  specialist  to  a  reference  in  the  FAR 
that  establishes  the  dichotomous  relationship  between  PBA  and  LOE.18 

Contracts  specialists  must  report  every  contract  in  FPDS.  Establishing  this 
conflict  in  FPDS  will  educate  contracting  officers  that  the  two  types 
cannot  be  used  together. 

•  Clarify  in  the  DOD  Guidebook  for  the  Acquisition  of  Services  that  PBA 
and  LOE  are  not  compatible.  The  dichotomous  relationship  of  PBA  and 
LOE  can  be  clarified  in  several  sections  of  this  document,  including  but 
not  limited  to:  1.5, 1.7,  2.8,  4.2,  4.3,  4.5,  4.6,  5.2. 1.2,  and  6.2.4. 


18  This  reference  can  be  37.602(b)(1)  or  a  new  reference  if  the  aforementioned  recommendation  is 
adopted. 
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This  document  is  intended  to  be  an  important  resource  for  the  DOD 
contracting  workforce;  this  is  a  logical  place  to  stress  the  incompatibility 
of  PBA  and  LOE. 

•  Review  policy  at  all  levels  to  identify  additional  opportunities  to  stress  the 
incompatibility  of  PBA  and  LOE.  There  are  many  policy  documents  at 
many  different  levels  that  address  the  use  of  PBA.  As  demonstrated  in 
Chapter  IV,  contracting  for  services  on  an  LOE  basis  is  often  overlooked. 
DOD  can  close  this  gap  by  reviewing  policy  and  identifying  additional 
opportunities  to  stress  the  difference. 

•  Review  Defense  Acquisition  University  (DAU)  and  Naval  Postgraduate 
School  (NPS)  course  material  for  opportunities  to  stress  the  PBA-LOE 
conflict.  DOD  offers  courses  on  contracting  for  services  through  DAU  and 
NPS.  The  course  text  for  NPS  Course  MN4311,  evaluated  as  part  of  this 
study,  did  not  discuss  that  PBA  and  LOE  cannot  be  combined.  Similar 
deficiencies  may  exist  in  DAU  course  material. 

•  Issue  a  policy  memorandum  to  inform  the  workforce  of  the  dichotomous 
relationship  between  PBA  and  LOE.  Although  USD(AT&L)  has  indicated 
that  PBA  and  LOE  are  two  distinct  choices,  it  has  not  done  so  directly  in  a 
stand-alone  memorandum.  This  study  revealed  that  the  Navy  often  applies 
PBA  inappropriately  to  LOE.  The  Navy,  or  perhaps  USD(AT&L),  should 
issue  a  stand-alone  memorandum  to  stress  that  PBA  and  LOE  are  two 
distinct  choices. 

2.  Eliminate  perverse  incentives  to  use  PBA  when  it  does  not  apply. 

•  Add  LOE  services  to  the  list  of  services  exempt  from  PBA  reporting 
requirements.  Contracting  activities  should  not  be  motivated  to  report 
LOE  services  contracts  as  PBA.  Establishing  the  exemption  will  allow 
contracting  activities  to  properly  separate  PBA  and  LOE  without  fear  of 
failing  to  meet  PBA  implementation  goals. 

•  Remove  DFARS  237.170-2(a),  which  requires  higher  level  approvals  of 
non-PBSA  contracts.  This  study  found  that  Navy  contracting  officers  were 
faced  with  a  difficult  dilemma — either  attempt  to  mask  a  services 
requirement  as  PBA  and  report  it  as  such,  or  submit  a  waiver  to  the  HCA 
for  every  complex  services  contract.  Removing  the  higher  level  approval 
requirement  will  allow  contracting  officers  to  appropriately  separate  PBA 
and  LOE.  The  Navy  has  effectively  removed  this  requirement  through 
NMCARS  Change  13-05,  but  it  should  be  removed  for  all  services. 

3.  Help  ensure  proper  performance  incentives  and  contract  surveillance  of 
LOE  contracts. 
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•  Adopt  a  standard  surveillance  plan  for  Non-PBSA  contracts.  As  suggested 
in  Chapter  IV,  proper  perfonnance  incentives  can  exist  in  LOE  contracts 
without  the  use  of  PBA.  Through  the  use  of  CPARS,  performance  can  be 
subjectively  evaluated  and  rated.  A  standard  surveillance  plan  could  be 
created  to  track  directly  to  the  CPARS  rating  elements.  An  example  of 
such  a  plan  is  included  as  Appendix  E. 

•  This  study  also  suggests  that  there  is  unnecessary  administrative  burden 
associated  with  creating  and  following  a  QASP  that  is  tailored  to  specific 
perfonnance  results  when  such  performance  results  cannot  be  identified 
(as  under  LOE).  By  adopting  a  standard  plan  for  Non-PBSA  contracts,  this 
unnecessary  administrative  burden  can  be  eliminated. 

C.  AREAS  FOR  FURTHER  RESEARCH 

The  sampled  contracts  in  this  study  included  only  those  awarded  by  the  Navy  and 
Marine  Corps.  Further  research  could  examine  how  often  PBA  and  LOE  are  combined  in 
other  government  agencies;  the  problem  could  be  pervasive  across  federal  government. 

The  study  found  that  LOE  contracts  outweighed  completion  contracts  in  the 
sample  (22  to  9).  Although  the  sample  was  limited  to  contracts  coded  in  FPDS  as  PBS  A, 
and  although  several  specifications  were  not  available,  the  data  still  suggests  that  LOE  is 
used  quite  frequently  for  the  acquisition  of  services.  Further  research  could  identify  how 
often  LOE  is  used  as  a  percentage  of  all  government  or  DOD  contracts. 

This  study  found  that  in  19  of  the  50  contracts  sampled,  it  was  not  clear  whether 
the  contract  was  completion  or  LOE.  As  discussed  in  Chapter  II,  this  was  either  due  to 
lack  of  access  to  the  specification  or  an  apparent  intermingling  of  completion  and  LOE. 
These  present  potential  problems.  Further  research  efforts  could  include  gaining  access  to 
specifications  not  available  in  EDA,  then  identifying  conclusively  how  often  completion 
and  LOE  are  intermingled.  Theoretically,  because  the  two  types  are  fundamentally 
different  in  how  they  obligate  the  contractor  to  perform  the  work,  intermingling  could 
make  the  rights  of  the  parties  ambiguous.  Further  efforts  are  needed  to  study  this 
potential  issue. 
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APPENDIX  A.  SAMPLED  CONTRACTS 


Referenced  IDV  Pll  ▼  j 

PHD 

Date  Signs  w  Type  of  Contract 

N0003314D8005 

1033 

6/26/2014 

COST  PLUS  FIXED  FEE 

N6833515P0019 

10/17/2014 

COST  PLUS  FIXED  FEE 

N0018911DZ041 

22 

9/25/2014 

COST  PLUS  FIXED  FEE 

N6893611D0005 

8 

2/5/2015 

COST  PLUS  FIXED  FEE 

N0017804D4119 

HR07 

3/19/2015 

COST  PLUS  FIXED  FEE 

N6523609D3808 

313 

8/22/2014 

COST  PLUS  FIXED  FEE 

M6785414C0015 

5/22/2014 

COST  PLUS  FIXED  FEE 

N0042114D0073 

12 

11/4/2014 

COST  PLUS  FIXED  FEE 

N0018904D0040 

8GT 

11/14/2014 

COST  PLUS  FIXED  FEE 

N6600113D0101 

25 

3/5/2015 

COST  PLUS  FIXED  FEE 

N0001911D0002 

139 

2/11/2015 

COST  NO  FEE 

N0042114D0002 

55 

9/5/2014 

COST  PLUS  FIXED  FEE 

N0002411D4115 

115 

9/15/2014 

COST  PLUS  AWARD  FEE 

N6660410D003A 

20 

8/26/2014 

COST  PLUS  FIXED  FEE 

N0001911D0002 

137 

12/23/2014 

LABOR  HOURS 

N6554013D0006 

22 

9/25/2014 

COST  NO  FEE 

N0002413D6400 

379 

8/12/2014 

COST  PLUS  FIXED  FEE 

N0042111D0040 

61 

7/29/2014 

COST  PLUS  FIXED  FEE 

N6523609D2700 

235 

4/15/2014 

TIME  AND  MATERIALS 

N6523613D7802 

74 

5/6/2014 

COST  PLUS  FIXED  FEE 

N6523609D3808 

315 

9/8/2014 

COST  PLUS  FIXED  FEE 

N5005414D1408 

24 

12/31/2014 

COST  PLUS  FIXED  FEE 

N6600109D0041 

48 

6/4/2014 

COST  PLUS  FIXED  FEE 

N6600113D0106 

31 

1/15/2015 

COST  PLUS  FIXED  FEE 

N6893612D0024 

EF02 

7/31/2014 

COST  PLUS  FIXED  FEE 

N0042115D0003 

1 

2/5/2015 

COST  PLUS  FIXED  FEE 

N6833514C0196 

4/30/2014 

COST  PLUS  FIXED  FEE 

N6554015D0014 

1 

4/2/2015 

COST  PLUS  FIXED  FEE 

N0017314D2004 

5 

5/1/2014 

COST  PLUS  FIXED  FEE 

N6523610D2839 

779 

6/5/2014 

COST  PLUS  FIXED  FEE 

N0016715D0003 

1 

2/24/2015 

COST  PLUS  FIXED  FEE 

N6600114D0121 

5/22/2014 

COST  PLUS  FIXED  FEE 

N6554010D0012 

23 

7/14/2014 

COST  PLUS  FIXED  FEE 

N0018911D0009 

81 

1/27/2015 

COST  PLUS  FIXED  FEE 

N6600114D0094 

10 

2/26/2015 

COST  PLUS  FIXED  FEE 

N0042113D0002 

52 

5/13/2014 

COST  PLUS  FIXED  FEE 

N6247011D8007 

8 

9/30/2014 

COST  PLUS  FIXED  FEE 

N0001411D0323 

501 

9/8/2014 

COST  PLUS  FIXED  FEE 

N6600114D0032 

1 

8/12/2014 

COST  PLUS  FIXED  FEE 

N0003911D0033 

100 

1/8/2015 

COST  PLUS  INCENTIVE  FEE 

N6554013D0008 

19 

7/23/2014 

COST  PLUS  FIXED  FEE 

N0018911D0009 

75 

4/25/2014 

COST  PLUS  FIXED  FEE 

N6274210D1804 

22 

6/17/2014 

COST  PLUS  AWARD  FEE 

N6523610D3824 

9 

3/9/2015 

COST  PLUS  FIXED  FEE 

N6600109D0014 

83 

9/30/2014 

COST  PLUS  FIXED  FEE 

N0042114D0002 

46 

7/15/2014 

COST  PLUS  FIXED  FEE 

N6258310D0342 

7 

7/31/2014 

COST  PLUS  FIXED  FEE 

N0001411D0323 

532 

9/23/2014 

COST  PLUS  FIXED  FEE 

N6523613D2800 

87 

2/25/2015 

COST  PLUS  FIXED  FEE 

N0018912DZ010 

133 

10/27/2014 

COST  PLUS  FIXED  FEE 

45 
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APPENDIX  B.  DETAILED  EVALUATION  CRITERIA  FOR 
DETERMINING  PROPER  APPLICATION  OF  PBA 


This  Appendix  contains  further  infonnation  regarding  how  the  researcher 
evaluated  the  minimum  PBA  criteria.  This  Appendix  lists  each  minimum  criterion 
followed  by  the  method  for  evaluating  the  criterion. 

A.  CRITERION  #1 

Criterion  #1  is:  The  PWS  describes  a  required  result  rather  than  either  “how”  the 
work  is  to  be  accomplished  or  the  number  of  hours  to  be  provided  (FAR  37.602(b)(1)). 
This  standard  was  broken  into  two  questions:  1)  did  the  specification  (whether  or  not  it 
was  labeled  as  a  PWS)  specify  a  perfonnance  result?,  and  2)  did  the  specification  specify 
the  number  of  hours  to  be  provided?19  Based  on  the  FAR’s  definition,  1)  and  only  1) 
must  be  “YES”  to  satisfy  this  criterion. 


1 .  Did  the  specification  specify  a  perfonnance  result? 

The  researcher  relied  on  the  following  assumptions: 

•  Reports  such  as  studies,  analyses,  status  reports,  document  revisions,  and 
plans  do  not  alone  meet  the  standard  of  “performance  results.”  Although 
these  reports  are  deliverables,  they  are  typically  by-products  and  are  not 
considered  to  be  the  required  outcome.  The  DOD  Guidebook  for  the 
Acquisition  of  Services  suggests  that  for  PBA,  the  focus  should  be  on  the 
“mission  outcomes  you  are  trying  to  achieve”  (DOD,  2012,  p.  10).  The 
Guidebook  also  suggests  asking  “WHY  is  this  action  needed?”  (DOD, 
2012,  p.  26).  The  requirement  to  deliver  only  reports  does  not  answer  the 
question,  “Why?”  Therefore,  contracts  requiring  only  reports  did  not  meet 
this  standard. 


19  Note  that  there  is  one  additional  question  that  could  have  been  asked:  Does  the  specification 
describe  “how”  the  work  is  to  be  accomplished?  However,  this  determination  would  be  subjective  so  it  was 
therefore  not  considered.  Without  greater  familiarity  with  the  specific  requirement,  the  researcher  was  not 
comfortable  determining  that  a  given  specification  was  either  too  prescriptive  or  not  prescriptive  enough  to 
be  classified  as  PBA.  As  discussed  in  Chapter  IV,  the  contrast  between  questions  1)  and  2)  were  sufficient 
to  arrive  at  the  conclusion. 
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•  Ad  hoc  services  or  deliverables  do  not  constitute  performance  results.  A 
given  requirement  may,  by  its  nature,  lend  itself  to  specifying  specific 
perfonnance  results.  For  example,  a  requirement  for  vehicle  maintenance 
could  require  that  “80%  of  vehicles  are  operational” — this  would  be  a 
clear  perfonnance  result.  However,  for  good  or  bad  reasons,  such  a 
requirement  may  be  described  only  as,  “The  contractor  shall  perfonn 
maintenance  on  vehicles.”  Without  a  clear  performance  result,  the  specific 
maintenance  activities  appear  to  be  ad  hoc,  at  the  direction  of  the 
government.  If  a  requirement  did  not  include  clear  perfonnance  results, 
this  standard  was  not  met. 

2.  Did  the  specification  specify  the  number  of  hours  to  be  provided? 

Although  this  information  was  not  always  contained  in  the  same  section  of 
the  contract,  it  was  fairly  easy  to  identify.  Locations  included  contract  line 
items,  special  clauses,  text  in  Section  B,  and  the  specification. 

If  1)  and  only  1)  was  “YES,”  meaning  that  the  specification  describes  a  required 
result  rather  than  the  number  of  hours  to  be  provided,  the  standard  was  met.  If  not,  the 
standard  was  not  met. 

B.  CRITERION  #2 

Criterion  #2  is:  The  contract  shall  include  measureable  perfonnance  standards 
(FAR  37.601(b)(2)).  Whether  or  not  the  contract  specified  a  performance  result,  the 
researcher  reviewed  the  specification  to  determine  if  it  included  measureable 
perfonnance  standards.  The  researcher  reviewed  the  standards,  if  they  existed,  and 
attempted  to  detennine  if  these  were  measureable.  The  detennination  of  whether 
something  was  “measurable”  required  creative  thinking.  For  example,  standards  such  as, 
“documents  are  technically  accurate  and  grammatically  correct,”  were  common. 
Standards  such  as  this  were  detennined  to  be  measureable,  because  in  theory,  one  could 
review  and  scrutinize  each  document  delivered  and  measure  by  counting  errors. 

The  researcher  first  reviewed  the  specification  for  this  information.  If  the 

information  was  not  contained  in  the  specification,  the  researcher  reviewed  the  QASP.  In 

the  vast  majority  of  cases,  the  QASP  was  either  not  mentioned  and  not  attached,  or 

mentioned  but  not  attached.  When  the  QASP  was  not  mentioned  and  not  attached,  the 

researcher  concluded  that  the  contract  did  not  include  measureable  performance 

standards.  One  may  contend  that,  because  QASPs  are  not  supposed  to  be  included  as  part 
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of  the  contract  (per  Section  4.3.3  of  the  DOD  Guidebook  for  the  Acquisition  of  Services), 
perfonnance  standards  may  still  exist  in  a  QASP  that  is  not  mentioned  and  not  attached 
(DOD,  2012,  p.  28).  However,  this  contention  would  ignore  the  need  to  have  high  level 
objectives  and  tasks  tied  to  standards  in  the  resultant  contract  (DOD,  2012,  p.  34). 
Providing  the  contractor  a  non-binding  QASP  without  formally  requiring  performance 
objectives,  standards,  and  acceptable  quality  levels  would  render  the  PBA  approach 
meaningless.  Therefore,  contracts  that  did  not  contain  measureable  standards  in  the 
specification,  did  not  mention  a  QASP,  and  did  not  have  a  QASP  attached  failed  to  meet 
this  standard. 

However,  when  the  QASP  was  mentioned  but  not  attached,  the  researcher  could 
not  conclude  whether  this  standard  was  met;  such  contracts  were  recorded  as  “not  clear” 
in  the  data  sheet.  As  discussed  in  the  previous  paragraph,  the  QASP  should  not  be  the 
only  document  that  contains  the  measurable  standards.  However,  because  the  QASP  was 
referenced  in  the  contract,  one  could  argue  that  such  standards  are  thereby  contractually 
required.  The  researcher  does  not  agree  with  this  position,  because  per  DOD  Guidebook 
for  the  Acquisition  of  Services  Section  4.3.3,  the  QASP  should  be  furnished  to  the 
contractor  merely  as  an  “informational  copy”  (DOD,  2012,  p.  28).  However,  the 
researcher  provides  the  benefit  of  the  doubt  to  those  that  would  argue  for  this  approach. 
As  discussed  in  the  Chapter  IV,  closer  evaluation  of  this  standard  was  not  necessary  to 
arrive  at  the  overall  conclusion. 

C.  CRITERION  #3 

Criterion  #3  is:  The  contract  shall  include  a  method  for  assessing  performance 
against  perfonnance  standards  (FAR  37.601(b)(2)).  The  evaluation  of  the  government’s 
method  for  assessing  perfonnance  required  less  creativity.  Simply  put,  this  standard  was 
met  if  the  government  identified  any  method  for  assessing  the  perfonnance  standards. 
Note  that  objectivity  was  not  assessed  at  this  stage;  this  is  discussed  in  Criteria  #4. 

The  researcher  first  reviewed  the  specification.  If  the  infonnation  was  not 
contained  within  the  specification,  the  researcher  reviewed  the  QASP.  The  DOD 
Guidebook  for  the  Acquisition  of  Services  Section  4.3.3  does  not  require  the  surveillance 
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method  to  be  stated  in  the  contract,  because  “this  enables  the  government  to  make 
adjustments  in  the  method  and  frequency  of  inspections  without  disturbing  the  contract” 
(DOD,  2012,  p.  28).  Therefore,  if  the  QASP  was  either  not  mentioned  or  not  attached,  the 
researcher  was  unable  to  detennine  whether  this  standard  was  met.  Such  contracts  were 
recorded  as  “not  clear.” 

D.  CRITERION  #4 

Criterion  #4  is:  The  contractor’s  perfonnance  against  the  required  standards  must 
be  measureable  through  an  objective  process.  The  evaluation  of  objectivity  was  based  on 
a  review  of  the  stated  method  of  assessing  the  perfonnance  standards.  The  researcher’s 
evaluation  of  this  category  is  best  described  through  two  examples.  The  standard,  “100% 
of  documentation  inputs  are  provided  by  the  due  dates,”  was  detennined  to  be  objective, 
because  one  could  count  the  number  of  inputs,  compare  them  to  the  due  dates  to 
detennine  how  many  were  on  time,  and  arrive  at  a  percentage  of  those  that  were 
submitted  timely.  Such  an  assessment  does  not  involve  subjectivity — an  input  is  clearly 
either  on  time  or  late. 

However,  assessment  methods  can  be  subjective.  For  example,  the  following 
standard  cannot  be  objectively  measured:  “Review  equipment  operation  procedures  to 
ensure...  safety  [is]  not  adversely  impacted  by  emerging  technology  dictated 
modifications.”  This  does  not  indicate  how  safety  will  be  objectively  measured.  Who  is 
to  say  whether  or  not  safety  is  adversely  impacted?  If  perfonnance  was  not  measureable 
through  an  objective  process,  this  standard  was  not  met. 

For  the  same  reasons  discussed  in  the  previous  standard,  if  the  QASP  was  either 
not  mentioned  or  not  attached,  the  researcher  was  unable  to  determine  whether  this 
standard  was  met.  Such  contracts  were  recorded  as  “not  clear.” 
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APPENDIX  C.  BLANK  DATA  SHEET 


Contract  Contract  Contract 
Number  Type  Number 

Contract  LOE  or 

PHD  Type  Completion? 

Specification  is 
identified  in  the  title 

as  a  PWS? 

Specifies 

Specifies  Number  of 

performance  Hours  to  be 

result?  provided? 

Method  of  Performance 

assessing  can  be 

performance  measured 

Includes  against  through  an  PBSA 

measurable  performance  objective  properly 

standards?  standards?  process?  applied? 

QASP 

mentioned 

but  not 

attached 

QASP  not 

mentioned 

or  attached 
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APPENDIX  D.  COMPLETED  DATA  SHEET 


Contract  Number  PHD  Contract  Type 

N0003314D8005  1033  COST  PLUS  FIXED  FEE 

N0018911DZ041  22  COST  PLUS  FIXED  FEE 

N0017804D4119  HR07  COST  PLUS  FIXED  FEE 


Specification  is  Specifies 
LOEor  identified  in  the  performance 

Completion?  titleasaPWS?  result? 

LOE  No  No 

LOE  Yes  No 

LOE  Yes  No 


Specifies  Number  of  Includes 
Hours  to  be  measurable 

provided?  standards? 


Method  of 

assessing 

performance 

against 

performance 

standards? 


Performance  can 

be  measured  QASP 

through  an  P8SA  mentioned  QASP  not 

objective  properly  but  not  mentioned 

process?  applied?  attached  or  attached 


yes  No 


N6523609D3808  313  COST  PLUS  FIXED  FEE  LOE  YES  No  Yes  No  ?  _  ?  No 

N0042114D0073  12  COST  PLUS  FIXED  FEE  LOE  No  No  Yes  No  ?  ?  No 


N0042114D0002 


55  COST  PLUS  FIXED  FEE 


N0002413D6400 


379  COST  PLUS  FIXED  FEE 


N0042111D0040 

N6523613D7802 
N6523609D3808 
N6893612D0024  EF02 
N0042115D0003 
N0017314D2004 

N6523610D2839 


N0016715D0003 


61  COST  PLUS  FIXED  FEE  LOE  Yes  No  Yes 

74  COST  PLUS  FIXED  FEE  LOE  yes  No  YES 

315  COST  PLUS  FIXED  FEE  LOE  Yes  No  Yes 

COST  PLUS  FIXED  FEE  LOE  No  No  Yes 

1  COST  PLUS  FIXED  FEE  LOE  [no _ [no  Yes 

5  COST  PLUS  FIXED  FEE  LOE  Wes _ Wes  Yes 

779  COST  PLUS  FIXED  FEE  LOE  Yes  No  Yes 

1  COST  PLUS  FIXED  FEE  LOE  Yes  No  Yes 


N66001 14 D0094  10  COST  PLUS  FIXED  FEE  LOE  Yes  No  Yes  yes  Yes  Yes  No 

N0001411D0323  501  COST  PLUS  FIXED  FEE  LOE  No  No  Yes  No  ?  ?  No 

N6554013D0008  19  COST  PLUS  FIXED  FEE  LOE  |no _ Mo _  Yes  No  ? _ ?  No 

N0042114D0002  46  COST  PLUS  FIXED  FEE  LOE  No  No  Yes  No  ?  ?  No 

N0018912DZ010  133  COST  PLUS  FIXED  FEE  LOE  yes  No  Yes  No  ?  ?  No 

N0001911D0002  137  LABOR  HOURS  n/a  No  spec  No  Yes  No  ?  ?  No 


N6523609D2700 

N6600113D0101 

N6660410D003A 

N0018911D0009 

N0003911D0033 

N0018911D0009 

N6523610D3824 

N0001411D0323 

N6893611D0005 

N0001911D0002 

N0002411D4115 

N6554013D0006 

N6554015D0014 

N6554010D0012 

N0042113D0002 

N6247011D8007 

N6274210D1804 

N6258310D0342 

N6523613D2800 

N0018904D0040 

N5005414D1408 

N6600109D0041 

N6600113D0106 

N6600114D0032 

N6600109D0014 


235  TIME  AND  MATERIALS 
M6785414C0015  COST  PLUS  FIXED  FEE 
25  COST  PLUS  FIXED  FEE 
20  COST  PLUS  FIXED  FEE 
N6600114D0121  COST  PLUS  FIXED  FEE 
81  COST  PLUS  FIXED  FEE 
100  COST  PLUS  INCENTIVE  FEE 
75  COST  PLUS  FIXED  FEE 
9  COST  PLUS  FIXED  FEE 
532  COST  PLUS  FIXED  FEE 
N6833515P0019  COST  PLUS  FIXED  FEE 
8  COST  PLUS  FIXED  FEE 
139  COST  NO  FEE 
115  COST  PLUS  AWARD  FEE 

22  COST  NO  FEE 

1  COST  PLUS  FIXED  FEE 

23  COST  PLUS  FIXED  FEE 
52  COST  PLUS  FIXED  FEE 

8  COST  PLUS  FIXED  FEE 
22  COST  PLUS  AWARD  FEE 
7  COST  PLUS  FIXED  FEE 

87  COST  PLUS  FIXED  FEE 

8GT  COST  PLUS  FIXED  FEE 

24  COST  PLUS  FIXED  FEE 

N6833514C0196  COST  PLUS  FIXED  FEE 

48  COST  PLUS  FIXED  FEE 

31  COST  PLUS  FIXED  FEE 

1  COST  PLUS  FIXED  FEE 

83  COST  PLUS  FIXED  FEE 


n/a 

Completii 

Completii 

Completii 

Completii 

Completii 

Completii 

Completii 

Completii 

Completii 
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APPENDIX  E.  SSC  PACIFIC  NON-PBSA  SURVEILLANCE  PLAN 


SSC  Pacific  Xon-PBSA  Sun  eiUnnce  Plan 


A.  Applicability 

The  following  surveillance  plan  is  applicable  to  all  service  contracts  and  task  orders  (including  Research  and 
Development  service  contracts  and  task  orders)  that  are  not  Perfonnance-Based  Sendee  Acquisitions  (PBSAs) 

B.  Purpose 

The  purpose  of  this  surveillance  plan  is  to  document  the  Contracting  Officer's 
Representative's  (COR's)  approach  to  monitoring  the  contract.  The  surveillance  plan  is 
attended  to  ensure  qualms  timely  delivery,  and  proper  cost  control  As  stated  m  the  COR 
Appointment  Letter,  the  COR  is  responsible  for  adhering  to  this  plan. 

The  stnveillance  plan  does  not  address  all  of  the  COR's  responsibilities.  Additional 
responsibilities  are  documented  m  SSCPACINST  4240. 1 .  the  SSC  Pacific  COR  Manual,  and 
the  COR  Appointment  Letter 

C.  COP.  Surveillance  Responsibilities 
The  COR  shall  perform  the  following 

(a) Revietv  Progress  Reports  (see  Attachment  1 .  Contractor  Report  Renew) 

(b) Review  Invoices  in  Wide  Area  Workflow  (see  Attachment  2,  Contractor 

Invoice  Review) 

(c) Review  Deliverables 

hi  addition  to  reviewing  die  progress  reports  discussed  above,  the  COR  shall 
review  all  other  deliverables  under  the  contract  including  hardware  prototypes, 
software  prototypes,  test  results,  final  technical  reports,  etc.  The  COR  shall  assess 
the  contractor's  conformance  to  contract  requirements,  specifications,  and 
standards  of  good  workmanship.  The  COP.  shall  document  die  review  of  contract 
deliverables  in  the  COR  file:  the  specific  format  of  this  documentation  is  at  the 
COR's  discretion. 

(d) Parhcipate  in  Meetings  (Formal  and  Informal) 

In  addition  to  the  required  duties  above,  the  COR  is  encouraged  to  meet  with  die 
contractor  both  formally  and  informally  to  the  maximum  extent  practicable  (e  g. 
kickoffs,  conference  calls,  demonstrations,  and  site  visits  when  posable)  Such 
communication  promotes  accurate  progress  information,  contractor 
accountability,  and  proper  contract  oversight  The  COR  shall  document  each 
meeting  m  the  COP.  file;  the  specific  fonnat  of  this  documentation  is  at  the 
COR's  discretion. 

(e) Review  compliance  with  the  SPAWAR  Service  Contracting  Tripwires  (only  if 

applicable). 

The  Tripwires  do  NOT  apply  to  contracts  awarded  to  Federally  Funded  Research 
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and  Development  Centers  (FFRDCs)  or  RDT&  E  services  with  a  Product  Service 
Code  (PSC)  beginning  with  an  "A"  and  ending  with  1. 2. 3. 4.  5.  or  7  If  the 
Tripwires  do  apply,  a  useful  checklist  for  review  is  available  through  the  SSC 
Pacific  2  0  website-  See  Page  2  of  the  "Contractor  Invoice  Review  Report"  (link 
below). 


https:  contracts  sscpac.nnici-navy.mil  Code20Resourcelubrary  Core 
cor  inst  inv  revievv.doc 
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ATTACHMENT  1  -  CONTRACTOR  REPORT  RE\TE\Y 


Subj  CONTRACT  NUMBER 
CONTRACTOR _ 

Ref:  (a)  DID  =  Type _ 

of  report  Period _ 

Covered  _ 


1 .  Reference  (a)  report  was  reviewed  on 


Quality  of  Product  or  Serv  ice 

ELEMENT  RE1 TWTD 

COMMENTS, DISCREP.ENCIES  /Required) 

Assess  the  contractors 
conformance  to  contract 
requirements,  specifications  and 
standards  of  good  workmanship. 

List  and  assess  any  sub-tasks  to 
indicate  different  efforts  where 
appropriate  For  example:  Are 
reports  data  accurate? 

Does  the  work  performed  meet  the 
specifications  of  the  contract? 

C  Yes 

□  No 

Does  the  contractor's  work 
measure  up  to  commonly  accepted 
technical  or  professional 
standards? 

□  Yes 

□  No 

Assess  the  degree  of  government 
direction  required  to  solve 
problems  that  anse  during 
performance 
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Schedule 

ELEMESTREUEnTD 

COMMENTSWISCREP.4XCIES  (Required) 

Assess  the  timeliness  of  the 
contractor  against  the  completion 
of  the  contract,  milestones,  delivers' 
schedules,  and  administrative 
requirements  (e  g.,  efforts  that 
contribute  to  or  effect  the  schedule 
variance).  Note  If  performance 
schedule  slippage  is  detected,  the 
COR.  should  determine  the  factors 
causing  the  delay  and  report  them 
to  the  Contracting  Officer. 

Cost  Control 

ELEMEST  REUEHED 

CO.\L\IEXrS.DI$CREP.LXCIES  (Rtqinred) 

Assess  the  contractor' s  effectiveness 
in  forecasting,  managing,  and 
controlling  contract  cost.  For 
example,  does  the  contractor  keep 
within  the  total  estimated  cost  (what 
is  the  relationship  between  the 
negotiated  cost  budgeted  cost  and 
actual  cost)? 

Did  the  contractor  do  anything 
innovative  that  resulted  m  cost 
savings1 

□  Yes 

o 

is 

□ 

Were  billings  current,  accurate  and 
complete? 

n  Yes 

n  No 

Are  the  contractor's  methods  for 
monitoring  the  budget  adequate? 

□  Yes 

C  No 
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Any  issues  identified  in  the  review  will  be  documented  in  wntmg.  attached  to  this  form 
and  forwarded  to  the  PCO  and  PM  as  applicable  with  a  copy  m  the  COR  file 

Conclusion  (Required  -  potential  performance  cost  problems): 


Contracting  Officer's  Representative  (Signature)  (Date) 
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ATTACHMENT  :  -  CONTRACTOR  INV  OICE  REMEW 


Date  of  review: _ 

Subj:  CONTRACT  NUMBER _ 

CONTRACTOR _ 

Ref:  (a)  Invoice  Number _ 

Invoice  Date _ 

Penod  Covered. 'Bill  in?  Cycle _ 

1 .  Reference  (a)  invoice  with  supporting  documentation  for  work  performed  under  subject 
contract  was  renewed  in  Wide  Area  Workflow  and  verified  against  the  supporting 
documentation  on _ 


2. 


ELEMEXT  REHE^TD 

COMMEXTS/DISCREPANCIES  (Required) 

Work  performed  meets  contract 
requirements? 

[1  Yes 

n  No 

Labor  Hours  Labor  Mix  reported 
are  consistent  with  the  work 
performed? 

□  Yes 

n  No 

Direct  Material  received  was 
validated,  considered  reasonable, 
and  determined  to  be  necessary0 

□  Yes 

□  No 

□  N/A 

Travel  validated;  considered 
necessary  and  not  in  excess  of 

Jomt  Travel  Regulations  (JTR) 
(COR  must  have  a  process  for 
reviewing  travel  sufficient  to 
ensure  costs  are  reasonable  and  do 
not  exceed  JTR)? 

□  Yes 

C  No 

r  n/a 
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Other  Direct  Cost  (ODC)  such  as 
incidental  material  travel  other 
non-labor  costs  (but  NOT 
subcontractor  or  consultant  costs) 
reviewed  and  considered 
reasonable  and  necessary? 

[3  Yes 

C  No 

n  N/A 

Contractor  Acquired  Property 
validated  and  determined 
reasonable  and  necessary? 

r  Yes 

n  No 

□  N/A 

Subcontractor  Labor  Hours  Labor 
Mix  reported  are  consistent  with 
the  work  performed1 

□  Yes 

□  No 

D  N/A 

Any  issue:  identified  in  the  invoice  review  will  be  documented  in  writing.  attached  to  this 
form  and  forwarded  to  the  PCO  and  PM  as  applicable  with  a  cops'  m  the  COR  file 

Conclusion  (Required  -  potential  performance  cost  problems): 


Contracting  Officer’  s  Representanve  (Signature)  (Date) 
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